Four demo scenarios, all live-verified on the test cluster (2026-10-19/20):
1. pend_stall diagnosis (most reliable; includes the change-triggered
second-fire follow-up demo)
2. real-LLM cross-field inconsistency catch (glm queue-accounting finding,
reproduced twice)
3. worker_health node-down detection (fastest; documents the repeat-outage
dedup gap found while verifying: lastKey is never cleared on recovery,
so a second identical outage stays silent — one-line fix candidate)
4. MCP scheduler_advisor_findings query (zero-risk closer, auditability)
Plus pre-flight checks, advisor start command, known pitfalls table
(in-memory API keys die on apiserver restart, glm timeouts degrade to
echo, SSE listener timeout vs 5-min stall threshold), and cleanup
checklist.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The advisor's findings previously lived only in its JSONL decision log on
its own host. P1 adds two observe-only outlets so diagnoses reach humans
and other agents live:
apiserver side:
- backend/broker.go: EventAdvisorFinding ("advisor.finding") event type +
AdvisorFinding struct + Event.AdvisorFindings field.
- controllers/events.go: ReportFindings — POST /xce/v1/advisor/findings
(generated_at/scenario/findings), publishes one broker event. Broadcast
only: the decision log remains the store of record.
- router/routers.go: route behind RequirePermission("advisor:report").
- middleware/whitelist.go: admit /xce/v1/advisor/findings in the POST URL
whitelist (the URL middleware 403s before the router's RBAC otherwise —
same shape as bug-1489).
- config/rbac.yaml: advisor_agent role (advisor:report + the six *:view
grants the snapshot needs) and advisor:report in the permission
whitelist. mcp_readonly stays strictly *:view. New
TestRBACYAML_AdvisorAgentRole_PinsReportSurface pins the role surface.
advisor side:
- internal/push: Pusher — best-effort POST of each round's findings, one
retry, nil/unconfigured/empty findings are no-ops; failures log and
never affect the observe loop.
- internal/agent: SetPusher wired after the decision-log append (the log
is authoritative; push is a fan-out).
- internal/config + cmd: push default-on to the same XCE_API_URL with the
same XCE_API_KEY (the key's role needs advisor:report); ADVISOR_PUSH=
off|false|0 disables.
mcp-server side:
- internal/advisor.go: lenient JSONL reader for the advisor's decision
log (ADVISOR_LOG env), tail-bounded, corrupt lines skipped.
- scheduler_advisor_findings tool: recent records with parsed findings;
friendly errors when the log is missing/unconfigured. Passive reader —
never talks to the advisor process.
Cluster verification (t1/t2/t3): redeployed apiserver closure, deployed
rbac.yaml, issued advisor_agent key. Forced-PEND job → pend_stall fired
on the real LLM (glm-5-3-flash, 19.4s, degraded=false) → 2 findings, all
evidence verbatim in the snapshot, unverified=false → SSE broadcast
captured on /xce/v1/events → MCP tool returned both records. Second PEND
job joining the stalled set fired a new decision within one round
(change-triggered semantics). Verification jobs killed; sanity job
submitted→ran→DONE; all daemons and 3 hosts healthy.
Design doc: architecture diagram gains the push branch; roadmap P1
marked done (CLI outlet not built).
Co-Authored-By: Claude Code <noreply@anthropic.com>
cJobToModel pre-filled main_reason_name with pendReasonName(main_reason),
so a detail-only record (main_reason 0, the allocator's ordinary shortfall
shape) arrived at rawJobToModel named "unknown(0)" — and the promotion that
derives the overall reason from the first detail row requires an EMPTY
name. The promotion's own tests passed because they build rawJobData
directly; the live C path never hit it. Found by the advisor P0
verification: jobs whose res_reasons named no_resource read
main_reason_name=unknown(0) over the API.
main_reason 0 now leaves the name empty for the promotion to fill; a
non-zero main reason keeps its mapped name unchanged.
Co-Authored-By: Claude Code <noreply@anthropic.com>
snapshot: list jobs with AllUsers + submit_time:desc + limit 200 — the
advisor must see every user's PEND jobs newest-first, or pend_stall sees
nothing on a shared cluster (the default list is the caller's own jobs,
oldest first).
detect: drop the per-scenario cooldown from the fire path. The original
cooldown-first order suppressed a CHANGED situation for up to 10 minutes —
on the live cluster a new job joining the stalled set went unreported
until the cooldown expired. Semantics now: an unchanged situation (same
dedup key) never re-evaluates; any changed situation fires immediately.
A thrashing situation (A→B→A) firing per change is honest reporting — a
flapping node or oscillating stalled set is real signal.
Verified on the t1/t2/t3 cluster: two forced-PEND jobs (unsatisfiable
select; 64-slot limit rejection) both diagnosed in the first scan, a
kill changing the stalled set re-triggered within one 30s round, and
identical situations stayed deduped.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two live-verification defects in the jobs-list path the advisor depends on:
1. The client never passed "sort", so the apiserver's default
unique_job_id:asc + limit returned the OLDEST jobs first — every
recent PEND job was invisible to any limited list. JobFilter.Sort now
forwards to the sort query param.
2. The apiserver's job list handler appends submit_user:eq:<caller> when
no submit_user filter is given (controllers/jobs.go), so even a
scope-all service key saw only its own jobs (0 rows on the live
cluster). JobFilter.AllUsers emits filter={submit_user:eq:all} to get
the cluster-wide view; an explicit Owner still wins.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Architecture rationale (LLM out of the hot path, perception/authority
split, uncalibrated confidence never gates), provider contract, detector
table, runbook-as-poor-man's-RAG, honest limitations of commodity LLMs,
and the P1/P2 path.
Appendix A records the TypeSafe Jev (System One) integration design for
future use only: single /v1/systemone endpoint, noul/choice/score
primitives, the pend_stall_v1 four-question pack and its Finding mapping,
confidence-gated act mode, jev-1.13 jaggedness risk. Deliberately not
wired in P0.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/agent: observe-mode loop — periodic scan + SSE job-event pull
(events only pull the next scan earlier, debounced 5s; job.finished
feeds exit_burst). Provider chain: primary (LLM) → echo fallback on
failure, recorded as degraded. VerifyEvidence firewall runs on every
LLM answer. Observe-only by construction: no write path reachable.
- internal/declog: JSONL decision log (scenario, trigger, provider, model,
latency, tokens, findings, raw output, state snapshot + digest,
degraded flag), size-rotated.
- internal/runbook: incident-pattern seed (embedded YAML) from the
cluster's real debugging history (bug-1494 stale config, IPC ABI skew,
bug-1513 rejoin crashes, bug-1521 reason race, first-fit bias,
dispatch-fail repend family), rendered into the provider prompt.
- cmd: advisor run (signal-handled observe loop) and advisor once
(single round, JSON report to stdout, no decision log).
- internal/config tests; full suite CGO_ENABLED=0 go test ./... clean.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/snapshot: compact cluster state via schedulerclient (nodes,
PEND top-N most-starved with parsed pending_reason, cluster sums, queue
garnish, recent exits). Field whitelists + stall-threshold filter keep
the state a few KiB. Module-local PEND-reason table mirrors
mcp-server/internal/pendreason.go (lift into schedulerclient is a P1
cleanup).
- internal/detect: pend_stall / worker_health / util_skew / exit_burst
triggers — threshold, then per-scenario cooldown, then dedup on a hash
of the situation (job set + reasons / node set) so an unchanged cluster
is never re-evaluated.
Co-Authored-By: Claude Code <noreply@anthropic.com>
P0 of the scheduler advisor (observe-only LLM ops copilot). New standalone
Go module src/frontend/advisor (CGO_ENABLED=0, mcp-server layout, replaces
schedulerclient). No LLM executes anything: providers turn scenario
snapshots into structured Findings only.
- internal/config: env-driven config (XCE_API_URL/KEY, ADVISOR_LLM_*,
ADVISOR_INTERVAL, ADVISOR_LOG); LLM unset is valid (echo-only).
- internal/provider: Provider interface + Finding contract (kind/severity/
summary/evidence/confidence/suggestion). Confidence is display-only —
never calibrated for LLM backends, never gated on.
- echo: deterministic rules provider (fallback, test baseline, yardstick).
- openai: OpenAI-compatible chat-completions (GLM/Qwen/DeepSeek/local
vLLM), strict JSON response_format, enum-validated findings, bounded
429/5xx retries with backoff, 30s default timeout.
- VerifyEvidence: hallucination firewall — evidence elements must appear
verbatim in the state snapshot or the finding is demoted (unverified,
confidence 0.2) while staying in the log.
Jev (TypeSafe System One) deliberately not wired: recorded in the design
doc (later commit) as a future provider behind the same interface.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The per-job MIT-MAGIC-COOKIE-1 that unlocks an -X job's X11 forwarding
port rode every job reply that asked for it: any job:view caller could
read another user's cookie and connect straight to the worker's
forwarding port (6100-6199), bypassing the xattach relay's ownership
check. The SSE job stream carried it too, since its events embed the
full JobInfo row.
- apiserver: stripX11Cookie drops the cookie from GetJobsInfo,
GetJobByIDs and the SSE stream for every job the caller does not own
and holds no job:control grant over — the same rule XAttachJob
enforces. The row itself stays visible; only the secret goes.
- xcecli: preflightX11 checks DISPLAY, the Xauthority file and the
xauth binary before the job is submitted, so a misconfigured display
fails in one second instead of dispatching a job nobody can watch.
Verified end-to-end on t1: owner reads the cookie, a group mate with
job:view(self_ug)+job:control(self) gets the row with the cookie
stripped, an unrelated user is refused at the row level; controllers
tests cover the matrix plus the SSE path.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Render -X jobs' windows on the user's own X server with zero client
install, mirroring the ssh -X experience: the job command starts with
DISPLAY pointed at a per-job TCP listener on the worker, and the user's
xcecli relays that listener's byte stream to their local X server.
- xcecli: new -X/--x11 flag (interactive=2 + exdata "x11":1). After
submit it fetches the per-job MIT-MAGIC-COOKIE-1, registers it in the
user's Xauthority, then bridges one xattach websocket to the local X
server address parsed from $DISPLAY (unix socket or TCP). The cookie
entry is removed when the session ends. Pre-flight checks report
missing DISPLAY/xauth before submit.
- apiserver: GET /xce/v1/jobs/:id/xattach relays websocket binary frames
to the worker's forwarding port (raw TCP, no ttyd). Close codes match
the ttyd attach (4000 unreachable, 4001 ended). The job-finished event
ends the relay. Teardown closes both hops as soon as either pump
exits, so a client disconnect propagates instead of deadlocking in a
blocking read.
- worker: XJOB jobs with exdata x11=1 skip VNC, listen on 6100-6199,
generate a per-job xauth file (/tmp/job_<id>_xauth, worker-side since
the worker has no session), export DISPLAY/XAUTHORITY to the command,
and bridge the app connection (loopback) with the relay connection
(non-loopback; roles classified by source address, not arrival order).
A 60 s no-client watchdog reaps jobs nothing ever attached to; the
v1 single-connection boundary refuses a second X connection loudly.
t1-verified per the plan matrix: window rendering (xclock/xeyes on
Xvfb), normal end -> DONE + xauth removed, no-client 60 s reap (137),
kill -> immediate teardown with no residue, -I/-I --browse regression,
daemon health, missing-DISPLAY error, second-connection refusal.
(--xjob VNC on t1 fails for lack of a vncserver binary; pre-existing.)
Co-Authored-By: Claude Code <noreply@anthropic.com>
An attach job (xcecli -I without --browse) that no client ever connects
to sat in RUN forever: ttyd only starts the job command on the first
websocket client, so with no client the command never runs, ttyd never
exits, and the job holds its ttyd, port and per-job worker indefinitely.
New exdata key "attach":1, set by the CLI for TTY jobs that are not
--browse and carried through the apiserver's extendDataMap to the
worker's transfer_info->attach_mode. For such jobs the worker starts a
1 s watchdog when the job process starts:
- no client attached within 60 s -> reap ttyd (SIGKILL to the monitored
tree), job EXITs with status message "No client attached within 60 s"
- client came and the command ran and exited -> give ttyd 5 s to stop
on its own (-Q), then reap: "Command exited; ttyd did not stop on its
own"
The reap lands in the regular process_exit_callback path (JOB_EXITED,
status reason, post-exec, worker exit), so kill and cleanup behave like
any other job termination.
Field-critical detail: uv_proc_get_pid returns the "/bin/sh -c ttyd ..."
wrapper pid, and sh does not reliably exec -- on the field ttyd runs as
a child of the wrapper. Judging the wrapper's children would see ttyd
itself as "the command has started" and never reap. The watchdog
resolves the real ttyd pid first (/proc/<pid>/comm walk down through
the sh wrapper, capped depth) and judges client activity by the real
ttyd's children; killing still goes through the monitored wrapper tree
so both topologies (exec'd or forked) are covered.
Web-viewed sessions (--browse / DISPLAY) are untouched: no attach
exdata, no watchdog -- ttyd stays until a browser connects or the job
is killed, as before.
Attach failure on the CLI side prints that the job will be cleaned up
by the worker if no client attaches within 60 s.
Tests: xcecli (attach flag matrix), apiserver controllers (attach
exdata end-to-end in ExtendData), worker native suites; live t1
cluster: normal attach completes DONE with no residue, unattached job
EXITs at ~60 s with the reason in the worker log and port released,
Ctrl-C-then-wait completes when the command ends, --browse job stays
RUN past 60 s and kills cleanly.
Co-Authored-By: Claude Code <noreply@anthropic.com>
waitForInteractiveURL polled PEND every 5s, adding ~4.4s of dead time
to every interactive submit (the job typically reaches RUN with its
session URL ~0.6s after submit once the apiserver applies the
controller's jobPidInfo push), and the RUN-but-no-URL-yet round waited
another 2s. Front-load fast 500ms polls (10 rounds) so a normally
dispatched job attaches within ~1s, then back off to 2s so a job stuck
in a long queue costs the server a query every couple of seconds
instead of a permanent 2 qps; the RUN race polls 500ms too.
checkJobStatus also failed the wait outright on any state other than
PEND/RUN: treat PSUSP like PEND (still queued), and USUSP/SSUSP/
TRANSIENT as keep-waiting states — the allocation is held, a resume
(or failover settle) may follow, and tearing down the attach on a
transient state is wrong.
Cluster-verified on t1: submit-to-attach-attempt 0.56-1.08s across
three runs (previously ~6s even after the jobPidInfo fix), full
submit/run/finish lifecycle clean, daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
handleNotifyData had no case for NotifyData_data_jobPidInfo, so the
controller's start-report push (apiserver_job_start_fun, fired from
HOOK_DETAIL_JOB_START the moment WAL_JOB_START commits) was silently
dropped. interactive_job_url then stayed invisible until the next
periodic rusage-driven job UPDATE re-delivered the full row — up to
10s later (job_rusage_collect_interval), the dominant chunk of an
-I --tty job's submit-to-first-output latency (measured ~6s on t1,
server-side work itself is ~0.6s).
Add OnJobPidInfo to the notification interface and DefaultHandler:
it merges the worker-reported pids/pgids and the exdata {"url": ...}
onto the already-stored job record (get-mutate-put, mirroring
OnJobFinished) and publishes a job.updated event. The CGO decode
side is factored into cPidInfoToRaw/pidInfoJSONB, shared with the
existing xce_job_t conversion.
Cluster-verified on t1: URL is pollable ~0.6s after submit
(previously ~11s), full submit/run/finish lifecycle clean, all
daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three attach-mode (-I) fixes, verified end-to-end on the t1 cluster:
1. Session hang on job finish (job_attach.go): the relay's client->worker
pump blocked forever in ReadMessage after the worker hop closed, so the
old wg.Wait()-before-Close teardown never ran and the client hung, then
died with an abnormal 1006. A broker watcher now subscribes to
EventJobFinished and on job end sends the 4001 policy close and closes
both hops, unblocking the pumps; the teardown follows normally. The
client (attach.go) treats 4001/1000/1001 -- and 1006-after-output as a
final safety net -- as a clean session end.
2. "<<Starting on t1 >>" printed repeatedly (submit.go): the status poll
printed the banner on every transition; now gated on a reportedStart
flag. The URL is also no longer printed in attach mode -- it only makes
sense for browse (-I --tty --browse) and DISPLAY jobs.
3. ttyd left behind after an attach job exits (xcecd_worker.c): a new
tty_stop_attach_session(), called from process_exit_callback for TTY
jobs, kills the per-job ttyd right away. This is insurance -- the sh -c
exec means uv_proc monitors ttyd's own pid, so ttyd is normally gone
before the callback fires -- but it guarantees no orphan when the
timing lines up badly. Web-viewed sessions are untouched: they keep
ttyd and output alive for later browsing, per design.
Co-Authored-By: Claude Code <noreply@anthropic.com>
-I without --xjob now means -I --tty: the submit attaches the interactive
session right in the calling terminal through the apiserver relay. The
previous bare -I mapping (interactive=3, readonly DISPLAY web view) had no
interactive input and gave the user no hint; --browse still opens the web
URL for TTY jobs and --xjob keeps the VNC path. The API's interactive=3
semantics are unchanged -- only this CLI's default remaps.
Help text updated: -I/--tty/--xjob descriptions and the long-help examples.
Co-Authored-By: Claude Code <noreply@anthropic.com>
xjob_start_vnc's four failure points (XCE_SERVER_DIR unset, pre-exec spawn
failure, DISPLAY export failure, port-wait timeout) called exit(1), taking
down the whole per-job worker with no user-visible reason and leaving the
job RUNNING forever (the master never learned of the death until its
instance reap path). It now returns int; the worker_job_callback call site
reports the standard failed-start pattern -- JOB_FAILED + status_message
"Failed to start VNC server (check worker log)" + worker_exit_flag so the
master reaps the instance and starts a fresh one for the next job.
xjob_stop_vnc is cleanup-only: its failures now log and return instead of
exit(1) -- a failed stop must not kill the worker or clobber the job's
exit code.
generate_url sized its snprintf with CMD_MAX_LENGTH (4096) while the url
buffer is URL_MAX_LENGTH (256). Latent since import: the old build's
non-fortified snprintf silently truncated, but any _FORTIFY_SOURCE build
aborts with SIGABRT (__chk_fail) on the FIRST interactive job -- found via
core dump after t1 bare -I jobs died ~1s after start ("*** buffer overflow
detected ***" in generate_url). Size both snprintfs with URL_MAX_LENGTH.
Verified on t1: -I --xjob on a host without vncserver fails in ~1s with
the reason in worker/master logs, worker exits cleanly, next batch job
completes; bare -I submits, dispatches, attaches via the relay and runs
until killed; daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Step 2 of the interactive-job UX work: `-I --tty` now attaches the session
to the submitting terminal by default instead of opening a browser.
Producer/consumer shape (user clusters cannot reach worker ttyd ports):
xcecli (-I --tty, raw terminal)
<-> GET /xce/v1/jobs/:id/attach (JWT auth + job:control ownership)
<-> ws://worker:ttyd-port/ws (ttyd)
apiserver (controllers/job_attach.go):
- new websocket relay endpoint, route gated by RequirePermission("job:control");
owner may always attach, others need a job:control scope covering the owner
- JWT accepted from the Authorization header or ?token= query param
(browser WebSocket API cannot set headers)
- bidirectional frame pump with policy close codes: 4000 = cannot reach the
session on the worker, 4001 = session ended
- the worker hop dials with the "tty" websocket subprotocol: ttyd 1.7.7
completes the handshake without it but then stays silent (verified against
the deployed binary)
xcecli (cmd/jobs/attach.go):
- attach client: dials the relay, raw mode via golang.org/x/term, ttyd
wire format (TEXT hello {"AuthToken",columns,rows}; binary frames
'0'+input, '1'+resize JSON, output '0' to stdout), SIGWINCH resize
forwarding, terminal restored on every exit path
- submit.go: --browse flag opens the web URL instead of attaching (TTY jobs
attach by default; XJOB/VNC keeps the browser path); the URL line is
always printed as a record
- waitForInteractiveURL no longer fails when the job is RUN but the worker's
start report (carrying the URL) has not landed yet -- poll and retry
Verified on t1: raw-client and real-xcecli attach round trips through the
relay (echo both ways), kill -> EXIT with URL retained, batch jobs unaffected,
all daemons healthy. Suites: meson 'apiserver backend' 1/1, CGO_ENABLED=0
controllers/router and xcecli tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The worker packs the interactive session URL (ttyd/XJOB) into the start
report's exdata as {"url": ...}; the controller stores it on
job->job.pidinfo->exdata and the capnp Job wire carries pidinfo (pidinfo@9,
JobPIDInfo.exdata@3) — but apiserver's cJobToModel never mapped pidinfo, so
models.JobInfo.InteractiveJobURL (declared only in the query-field whitelist
JobAdditionalFields) had no producer and interactive_job_url was permanently
empty in API responses. xcecli's waitForInteractiveURL/openBrowser flow could
never fire.
- models.JobInfo: add InteractiveJobURL (json interactive_job_url, matching
the JobAdditionalFields whitelist name).
- convert_pure.go: rawPidInfo (pids/pgids/exdata) + interactiveURLFromExdata;
rawJobToModel marshals pids/pgids into PIDInfo (pid_info is a JSONBJFields
entry, parsed by processJobData) and promotes the exdata URL to
InteractiveJobURL.
- convert.go: cJobToModel extracts job.pidinfo via pidinfo_pid/pidinfo_pgid
cgo helpers (CGO cannot index the C arrays directly).
Cluster-verified end to end on t1 (fresh apiserver deployed): tty job
(xcecli -I --tty equivalent, exdata {"interactive":1}) now returns
interactive_job_url=http://<worker>:9000 and a populated pid_info; non-tty
jobs return ""/no pid_info. All daemons healthy, zero errors in window.
Co-Authored-By: Claude Code <noreply@anthropic.com>
test_sim_reconcile_unaccounted.c captures stdout through mkstemp() files in
/tmp/xce_simreconcile_*, but only the shared capture_stdout_around() helper
unlinked its file after reading it back -- the 4 inlined capture sites left
theirs behind on every passing run (the mkdtemp shim does not intercept
mkstemp, so nothing else removed them). Add the same close(fd); unlink(tmp)
the file family's other capture helpers use to each of the 4 sites.
Verified: scheduler sim reconcile unaccounted test passes and leaves no
/tmp/xce_simreconcile_* files behind.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).
Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.
Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
queue refuses new submissions only; pending jobs still run) --
supersedes our QUEUE_CLOSED preason on the dispatch path;
sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
rounds (sched_stash_config/sched_apply_pending_config), applied at
the same main-thread round boundary that preserves bug-1513
serialization
- bug-1760: sched_check_pending_rejoin exposed +
xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
(superset of our bug-1500 execuser fallback; our job-typed
resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
src/harness/test.c; controller suite split into
controller_test.c + fixtures; their clitest fixtures spawn xce_test
mid-suite under `meson test -j`
Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
gate re-ported into their relocated allocator (sched_election.c)
after plugin_job_election, with attach on the allocator thread
only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
merge initially re-carried our first-detail-add-promotes-main_reason
fallback in add_job_preason (4e2ac86), but their whole trace/verify
stack treats "detail-only => main_reason 0" as a contract --
sched_trace_capture counts main_reason as its own occurrence
("Expected 2 Was 3" in test_trace_capture), the outcome renderer
prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
and the committed .replay/.sim_verify baselines encode main_reason=0
(test_repl_trace_baseline byte compare, test_trace_equivalence
off-vs-traced). The fallback is dropped from the shared lib;
the user-visible "overall reason" survives as an API-layer view:
rawJobToModel (convert_pure.go) promotes the first detail row to
main_reason_name when main_reason is 0 (main_reason VALUE marshals
unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
(their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
resend, xcecli_force_reconnect + their is_connection_broken both
exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
/xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)
bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
(testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
harness_test.c -- hundreds of cases per process) call
case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
dirs are swept immediately in tearDown (after engine reaping), FAILED
cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
disables. One full controller_unit_test run was observed to leak 12G+
to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
failures
- harness_unit_test_deps links the shim library as well
Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
detail-keeps-main-0 contract; new tests cover limit-code detail
rows and explicit-main-wins-over-later-details; apiserver
convert_test.go gains the detail-only name-promotion cases
Co-Authored-By: Claude Code <noreply@anthropic.com>
A job without a project and without a group default goes to the cluster
default project, the built-in `default` (members all) unless configured
otherwise; it is never rejected for lacking a project.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A share may be owned at a location (host group) as well as anywhere.
Per resource, located host groups and the rest of the cluster form
disjoint pools; usage is charged to the host's pool and every pool follows
the share rules on its own, so located ownership is real. Pool capacity
counts registered hosts including down and closed ones. No-default
submits list the user's projects; a per-user default project is a
client-side xcecli setting; members see their ancestors' usage, not
siblings' details.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- `jobs` is a built-in resource for shares and member quotas: the job
limits per project and user; pending caps go to pending-job-limits.
- Jobs may be charged to inner projects, with a warning and pend reason
when their unassigned part is empty.
- Project aliases keep jobs through renames; running jobs follow them.
- Changing a pending job's project is part of job-modify.
- New project-accounting brief: durable finished-job records, outside the
WAL, before controller-job-cleanup drops a job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fairshare weights and usage history live on the project tree; queues keep
priority between queues and only choose fairshare or FCFS ordering. The
per-queue `fairshare:` user lists in queues.yaml go away.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A project's `share` (absolute or a percentage of its parent's) is carved
from the parent; siblings never add up to more than the parent. v1 is
strict: a project holds at most its share. A project's own jobs and its
children without a share draw from its unassigned part. Borrowing idle
shares above a project's own (`max`) and reclaim are future work, in the
renamed project-borrowing brief.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Revocation and expiry close live connections authenticated with the key,
since authentication happens only at join. Lists the design questions:
where the trusted list is changed, keeping revoked keys revoked, bootstrap
keys, token registration for elastic hosts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Per-host identity without a CA; only public keys move, and the trusted
list can ride on the replicated cluster config. Recommended over the
phased per-role-keys/PKI plan; decision still deferred.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Promotes the roadmap item to a brief: production/test mode on the
controller, authenticated roles at join, per-role request allowlist with
user requests only from the apiserver, an admin role for xceadmin. Lays
out the open decisions on secret type (cluster key / per-role keys / PKI)
and key distribution (files / token enrolment), with a phased
recommendation. project-config Req 3.13 applies to test mode only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Production submits always go through the apiserver; harness and direct
client-library submits are test tools. The apiserver resolves the project
and checks project:submit; the controller records the project and the
authorization basis it is given. Direct submits are recorded as given
with basis `direct`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Submitting under a project becomes the RBAC permission project:submit,
implied by membership (that project only) and grantable by roles (scope
covers the subtree). A project's admins get every job permission and
project:view over its subtree; they cannot edit the project, which only
the cluster admin changes through config in this stage.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The project tree is the only place concurrent use is capped: quotas are
written on the project (quota, member_quota over the subtree) and compiled
into the limit plugin, which becomes internal. Queues keep per-job
ceilings only; host slot caps are supply. Project path is the identity
and the only submit form; members are users and user groups, never
sub-projects. Adds the limits-to-owners backlog item.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Preset routes need no RBAC permission; a submit made from a preset is
checked like any other submit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1: any queue:submit user may use any profile; listing reuses cluster:view
- app:use / app:view design kept in the brief and moved to the new backlog
item app-access-control (depends on rbac-persistence for API grants)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- worker-run hooks: pre/post_job_exec (once, first host) and
pre/post_host_exec (once per host); no per-task hooks
- task starter reaches MPI ranks through the mpirun line XCE builds
(correction); XCE_JOB_HOST_DIR, XCE_HOST_NTASKS, XCE_HOST_TASK_INDEX
- hook failure: job EXIT with a reason naming the hook; post hooks still run
- post hooks must survive every way a job ends: run limit is systemd
RuntimeMaxUSec stopping the unit (worker included) and post-exec is only
bounded by the unset TimeoutStopUSec; requirement for an explicit stop
timeout and OOM survival (not logged as a bug: post-exec does run today)
- default profile always usable with queue:submit; service profile
rejected for batch jobs; warning when no host matches the profile select
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- params: declared application settings XCE carries but never interprets;
exported as XCE_APP_PARAM_*; job overrides with --app-param; undeclared
names rejected; stored in exdata app_params
- annotations: metadata for tools, also exported as XCE_APP_ANN_* (name
conversion, collision check, size limits, no secrets)
- merge drops 0-valued rusage terms so the scheduler sees no term (today a
0 term still rejects hosts where the resource is over-committed)
- app-submit-hook (esub-style) recorded as a roadmap backlog item only
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gaps closed:
- profile env travels separately from the job env (needed for the clean
outer-layer environment; remote tasks get both)
- merge library parses and rewrites exdata (env, limits; keeps meta);
added to job-config-merge scope
- new start_info fields listed (job starter, profile env)
- empty profile name = no profile, never resolved to default_application
- a file profile named default replaces the built-in one
- restart copies the profile name; array jobs carry one name
- multi-host outer pre/post-exec and job starter placement as open design
questions; rusage=0 and declined-dispatch counting to confirm
- scheduler-side field completeness (frozen dispatched jobs, Conf4Sched
profiles in snapshot/replay)
- no secrets in profiles; xceadmin warns on secret-looking env names
Stale text fixed: job-modify reference, exdata wording, rusage defaults
exception, I/O directories (moved to job-sandbox-execution), sandbox in the
sketch, job starter and job_pre_post_exec in the table, dispatch record
contents, misplaced rationale paragraph, versioning reference, scope and
touchpoints, task range (min/max) in job-config-layering.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile resreq split into select / rusage entries / span / same / order;
rusage entries carry optional per: (default task, today's PER_DEFAULT
behaviour) and optional default
- job's rusage term wins whole for a resource in both; unit must match;
no modifier inheritance
- compound/alternative: only the profile select applies; any profile
rusage default or span/same/order rejects until PR-15
- job-config-layering: merge rules at a glance table; profile = essentials
plus default reservations
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- rusage: union of profile and job resources; for a resource in both the
job's value wins (either direction); the profile's per-type applies and an
explicitly different per-type is rejected; omitted modifiers inherited;
name=0 means none
- rusage is the one profile section that is a per-resource default;
shared-resource protection via limits.yaml/quotas, later a counted marker
- side effects recorded for requirements (lowering warnings, redundant
preset values, origin labels, ceiling errors, hash churn)
- compound/alternative: profile select ANDed into every part, per-task
rusage merged per part, other profile resreq sections reject until PR-15
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- no preset field on the server: the client puts preset name + hash into
the job's exdata JSON under 'meta', passed through uninterpreted; the
worker skips 'meta' silently; 'preset report' filters jobs client-side
- drop the local submission log idea
- job-labels: independent backlog item (general server-side labels)
- bug-1970: note the dropped REST 'tags' field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1 profile attributes: mode (direct), launch type, env + pre/post-exec,
essential resreq and limits, plus the job starter
- principle: a profile attribute is added by the spec that builds its job
feature; requeue/success exit values -> job-requeue (new backlog item),
checkpoint and warning-before-kill later, output path -> I/O track,
runtime estimate is not a profile attribute
- log bug-1970: REST submit accepts rerunable, requeue_error_values,
run_time_estimation, mig_threshold, chkpnt_*, warning_* and drops them
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- server presets live in <TopDir>/config/presets/ (or presets.dir in
app.yaml), one file per preset, re-read on change; not in ClusterConfig
- pull refuses a name collision unless it updates the same server preset;
--as / --force (keeps .bak); submit uses local presets only
- fix route prefix: /xce/v1/presets, not /api/v1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
