- profile gone or changed between scheduler decision and controller
dispatch: dispatch fails (retryable) and the job returns to PEND via
requeue, never EXIT
- internal per-profile content hash (canonical capnp, excluding
description) carried on the dispatch decision and the dispatch record
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles hold only what every job of the application must have; every
profile value binds and a job may add to it, never change or remove it
(replaces the default/fixed markers of 2026-10-09)
- convenience defaults move to client-side presets (new xcecli-presets
brief): option-level expansion on the client, the apiserver only shares
admin-deployed presets read-only, pull copies, export/import
- service section stays in the profile: probes, restart policy, ports bind;
replica count within the profile range; spread/rollout if unset
- job-config-layering, job-config-merge brief/requirements, roadmap and
resource-questions updated to the additive rule
- app:override and profile inheritance deferred; recommendations target
site presets
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles exist mainly so users need not repeat resreq and other settings;
most values are defaults, only what the application needs is fixed
- usage-based recommendations are advisory (user decision); new backlog
item profile-usage-insight
- open questions: per-term resreq markers with ranges, app:override,
inheritance, presets, job labels, compound resreqs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles in src/configs/applications.yaml (+ schema, ClusterConfig and
Conf4Sched), optional file with a built-in 'default' profile
- default_application in xce.yaml; an undefined name is a config error at
xceadmin build, live upgrade and controller start, never a fallback
- remove the superseded submit-time resolution text: the job keeps its
profile name and own values, effective values come from job-config-merge
- this spec owns the profile attribute schema and override markers
- record the 2026-10-12 open questions with leanings
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LLM training is one parallel job with topology requirements (PR-10).
gang-groups moves from the spec list to the roadmap backlog, re-scoped as
a co-start group of separately managed jobs and services (RL
post-training, disaggregated serving, streaming pipelines). resource-model
phases and the brief updated; job-resize no longer resizes gang groups.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Persistent log of the resource-design discussion: the user's proposals
(generic supply map incl. provisioned supply; a resource provider proxy
between the controller and upstream providers; datasets in the map with
staging as data provisioning) with analysis, the user's 2026-10-11 input
(duration/decay must-have but future; RBAC in the apiserver, discuss later;
OCI containers via application profiles; CSI compatibility preferred;
GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items
and new questions N1-N4, C3. resource-model.md, the provisioner research,
roadmap and CLAUDE.md point at it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
resource-model.md: one resource type descriptor and provider contract over
three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler
only selects from its cached snapshot, providers run after dispatch; batch
stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims
as WAL entities with the resource-provisioner discipline; LSF/K8s mapping;
LLM training/serving needs; phases; nine pending user decisions.
New briefs: custom-resource-reporting, resource-instances,
resource-claims, gang-groups. Roadmap: long-term LLM training + serving
platform direction (features from LSF and Kubernetes), the four new specs,
updates to ten existing specs, discovery note. resource-distribution-mapping
becomes phase 1 (shared pools + descriptor fields).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scaler is multi-threaded (monitor and demand threads); invoke() forked
and, in the child, logged, snprintf'd and setenv()'d before execl(), which
can deadlock on a lock another thread held at fork time and block the
caller in waitpid() forever.
run_script() now posix_spawn()s `sh <script> -f <in.json>`: stdout and
stderr go to the output file through file actions, the provider variables
(PRO_CONF_DIR, PRO_DATA_DIR, PRO_LOG_DIR, PROVIDER_NAME) are passed as the
child's envp (provider_env()), the child starts with an empty signal mask
and default dispositions, and waitpid() retries on EINTR.
The scaler unit suite (scaler/testsuite) was never built; it is now the
meson test "autoscaler test", linked with -Wl,--wrap=fork so it counts
fork() calls during a provider call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
RestartJobByID rebuilt the new job from the old job's row:
- it accepted asked_hosts and copied asked hosts, I/O paths and out_dir,
which no job gets (refused at submit since bug-1936); restart now
refuses asked_hosts with 400 and no longer carries those over;
- it read the JSONB columns chkpnt_info/extend_data as strings, but rows
keep models.Jsonb, so env/limits were lost and a checkpointed job never
restarted from its checkpoint; new rowJSON() reads both;
- it read "res_req" (the column is "resreq"), losing the resource
requirement, and never copied exec_home.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Design constraint (tech.md "Process Creation"): the controller, scheduler,
xcecd and anything that starts libxce threads never call fork, vfork,
forkpty, daemon, system, popen or uv_spawn; children start with posix_spawn()
or a systemd unit. A process proven single-threaded may fork, marked
/* xce-fork-ok: <why> */. Tests, harness, autoscaler (not finalized; bug-1969)
and the Go frontend are out of scope.
- scripts/check-no-fork.py scans src/{controller,scheduler,lib,worker}
(comment/string/#if 0 aware); runs as meson test "no-fork rule" in the
lib/controller/scheduler/worker suites and in the pre-commit hook.
- worker: drop xcecd's unused fork-based worker start paths
(process_manager_start_process, process_manager_start_worker; xcecd
starts xcecd_worker over D-Bus) and a dead #if 0 daemonize fork.
- xcecd_worker's fork/uv_spawn/system calls are marked: its call graph
from main(), callbacks included, reaches no pthread_create or libuv
thread-pool call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: CreateJob accepted in_file, out_file, err_file, out_dir,
asked_hosts, begin_time, term_time and transfer_files and returned
success, but neither the controller's submit data/start info nor the
worker carries any of them (the worker never stages transferred files),
so they were silently dropped. Exec home had no submit field at all, and
a relative cwd would have been resolved against the worker unit's
directory.
Fix: new preCheckSubmitOptions() runs right after binding, before file
uploads and ID allocation, and answers 400 naming every unsupported
option that is set; cwd and the new exec_home field must be absolute and
fit the worker's job_assignment_t buffers (1023 bytes). exec_home goes
to JobInfo.ExecHome and from there (submitStartInfo) to the start info's
exechome. num_files already reaches exdata as nofile_limit; a handler
test now pins it. The array %J/%I submit-time substitution test (9.4)
checks cwd instead of out_file/err_file, which are now refused. Swagger
regenerated with `swag init -g cmd/main.go -o docs`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the start info's execcwd/exechome reached xcecd and its worker-list
entry, but the MSG_JOB_ASSIGNED reply to the per-job xcecd_worker
(job_assignment_t) carried neither, and uv_proc_exec() spawned the job
with cwd NULL, so every job ran in the xcecd_worker unit's directory
whatever --cwd said, and exec home was never applied. parse_limits()
ignored the apiserver's "nofile_limit", so num_files never became an
rlimit.
Fix: job_assignment_t gets cwd/home, filled by the new
ipc_fill_job_assignment() (uv_ipc.c, which also NUL-ends every copied
string). xcecd_worker's job callback calls the new
uv_proc_enter_job_dirs() (uv_proc.c) before pre-exec and the job: it
exports HOME and chdir()s into cwd, falling back to /tmp with a warning
when the directory cannot be entered (the xcecd_worker runs as the job
user, so this is the user's own access). parse_limits() reads
nofile_limit into res_limits_t.nofile_limit and start_systemd_job() sets
LimitNOFILE like LimitNPROC. xcecd_virtual spawns no job and is
unaffected.
Tests: testsuite/test_job_dirs.c (assignment carries cwd/home, enter
cwd/home, fallback, a spawned job's pwd is the cwd) and
test_apiserver_nofile_limit_parsed in test_exdata_parse.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller sends job control to every host of a multi-host job
(bug-1942) as soon as the job is dispatched, but a remote host's runtasks
request is relayed by the controller (task.c do_runtasks, stateless) only
after the head worker has started and asked for it. A kill/stop that
reached a remote host first found no worker and no pending start, was
dropped ("Failed to find worker for job ID"), and the task started later
and kept running.
pending_signal now also holds "early" entries: handle_jobcontrol() records
a signal for a job with no task here (pending_signal_record_early), and
handle_runtasks() adopts it (pending_signal_start_task), so it is delivered
at RUNNING through the bug-1908 path and the task's exit is reported as
usual. Early entries are bounded (256) and expire after 60 s, pruned on
every early record and task start. A late control for a task that already
finished here (finished-job retention) is still dropped, and a head-host
start does not adopt an early entry, so neither can hit a later run of the
same jobid.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
limit_seed_config_impl() key_add()ed each pushed limit into the existing
table and returned at the first failure, so on any second push (live
config upgrade, or a simulator CONFIG_PUSH since bug-1964) the first
already-present name failed with -EEXIST: changed limits kept the old
definition and later new limits were never added.
A push now builds the whole new limit set into a fresh table and swaps it
in (all-or-nothing: on any build/register failure the previous set and its
instances stay). Usage instances are carried across: an instance of a kept
limit whose scoping and resource are unchanged is rebound to the new
definition; an instance of a removed limit, or of one whose per-consumer
scoping, resource or percent changed, is dropped and re-created on the next
match. Usage itself lives in the resource manager under the instance's
virtual resource, so counts survive either way. Not part of the snapshot
(a snapshot carries the config and reseeds from it).
The seed moves to plugins/limit_config.c (limit.c is near the 800-line
cap) with the private data type in plugins/limit_internal.h. Two REPL
limit-table fixtures relied on the additive behaviour (two single-limit
pushes) and now configure both limits in one push.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: parjob's allocation now draws candidate hosts from the shared
sched_host_iter_next() (host_iter.c) instead of its own
sched_collect_workers() loop, so skip_bad_worker_ lost its only caller
(the compiler warns it is unused). seqjob's copy was already removed when
it moved onto the shared iterators (b18c5f78).
Fix: remove it. Its only check (a NULL worker) is done by
sched_host_iter_next(), which is also where host filtering now lives
(closed hosts, bug-1939; unavailable and reconciling hosts, bug-1963).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: live applies every pushed config through sched_init_config(),
whose plugin_config_update() runs each plugin's config_hook (the limit
plugin builds its limits there). The simulator seeded the limit plugin
only at bootstrap (sim_bootstrap_limit_plugin()); the core's CONFIG_PUSH
event loaded groups, resources and queues but never told the plugins,
so a limit added during a replay never existed in sim.
Fix: one shared fan-out, plugin_config_update_core(core, config), which
live's plugin_config_update() now delegates to. The core's CONFIG_PUSH
calls a driver-registered xce_core_config_cb_t after loading groups (live
order), keeping the core free of plugin.c; the simulator registers the
fan-out in sim_bootstrap_limit_plugin(). limit_update_config() reads the
honestly-typed plugin->core instead of casting plugin->sched, so the hook
is safe on the sim core; the hand-built plugin fixtures in
test_limit_plugin.c/test_repl_limit_table.c now set plugin->core as
plugin_register() does. test_sim_limit_config_push_parity.c covers it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: sched_host_iter_next(), the one candidate-host source of the
seqjob and parjob allocators on both drivers, skipped only admin-closed
hosts (bug-1939). A host marked XCE_WORKER_UNAVAIL or
XCE_WORKER_RECONCILING was still offered, and its frozen resInUse
(bug-1799/1801) does not reject it, so new jobs were placed on it.
Fix: skip IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING hosts next to the
closed-host check. test_host_unavail_reconciling.c covers serial and
parallel jobs on the normal-mode and sim-mode fixtures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since the controller rejects submissions to closed or unknown queues
(E_XCE_QUEUE_CLOSED, E_XCE_BAD_QUEUE), the apiserver answered them as a 500
"xcecli_jobsubmit failed: -10016". SubmitJob now maps them to
backend.ErrQueueClosed / ErrQueueInvalid (numbers checked against errnox.h
at compile time), and the submit and restart handlers answer 400 naming
the queue, as the apiserver's own pre-check does. This covers a queue
closed between that check and the controller's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: handle_jobsubmit() never looked at the target queue, so a
submission that bypassed the apiserver's check (libxce, harness
xce_test, or a REST submit without a queue name) was accepted into a
closed queue, an unknown queue, or the reserved lost_and_found queue.
Fix: jobsubmit_check_queue() reads the queue's in-memory state under the
qmgr/queue read locks. handle_jobsubmit() calls it after the field
checks and before the in-flight jobid registry and any WAL append, for
single jobs and job arrays. It replies -E_XCE_QUEUE_CLOSED (new code,
"queue is closed") for a closed queue and -E_XCE_BAD_QUEUE for an
unknown or reserved one. Apply/replay (restore_job) is unchanged, so
committed submit facts for a now-closed queue still apply.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the apiserver and all components ran go test on a fixed list
(./store/... ./utils/... ./controllers/... ./backend/...), so the
middleware, router, cmd and models tests (several security tests among
them) never ran in /xce-build; a deliberately broken middleware test left
the build green.
Fix: run ./... (vendor/ is excluded by the pattern) in both places.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the handler annotations referenced types that do not exist
(models.Error, ErrorResponse, JobsResponse, JobSignalRequest,
JobControlResponseSummary, unqualified JobSubmitRequest), had malformed
@Param/@Failure lines and /api/v1 paths, and there was no general API
info, so `swag init -g cmd/main.go` failed and docs/ stayed a stale stub.
Fix: use the real response envelopes (JsonStruct / JsonErrStruct) and
request models, the real routes under @BasePath /xce/v1, add general info
to cmd/main.go, and regenerate docs/ with `swag init -g cmd/main.go -o
docs`; the spec now has /jobs/{jobids}/control and /jobs/control.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: preCheckJobRequest (controllers/jobs.go) ran the reserved-name,
existence and closed checks only when a queue name was given; an empty
name was set to "normal" and skipped them, so jobs were accepted into a
closed default queue.
Fix: default the name first, then run the same checks for every
submission. Two existing tests now seed the default queue. (The controller
side of bug-1960 is fixed separately.)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: isExcludedPath (middleware/validatehandler.go) dropped the last
character of every excluded entry and prefix-matched, so /ping excused any
path starting with /pin (e.g. /pinX) from JWT validation, while
/swagger/*any (trimmed to /swagger/*an) never matched.
Fix: entries match exactly; an entry whose last segment is a gin catch-all
(/swagger/*any) matches any path under the part before the "*".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
