- profile resreq split into select / rusage entries / span / same / order;
rusage entries carry optional per: (default task, today's PER_DEFAULT
behaviour) and optional default
- job's rusage term wins whole for a resource in both; unit must match;
no modifier inheritance
- compound/alternative: only the profile select applies; any profile
rusage default or span/same/order rejects until PR-15
- job-config-layering: merge rules at a glance table; profile = essentials
plus default reservations
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- rusage: union of profile and job resources; for a resource in both the
job's value wins (either direction); the profile's per-type applies and an
explicitly different per-type is rejected; omitted modifiers inherited;
name=0 means none
- rusage is the one profile section that is a per-resource default;
shared-resource protection via limits.yaml/quotas, later a counted marker
- side effects recorded for requirements (lowering warnings, redundant
preset values, origin labels, ceiling errors, hash churn)
- compound/alternative: profile select ANDed into every part, per-task
rusage merged per part, other profile resreq sections reject until PR-15
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- no preset field on the server: the client puts preset name + hash into
the job's exdata JSON under 'meta', passed through uninterpreted; the
worker skips 'meta' silently; 'preset report' filters jobs client-side
- drop the local submission log idea
- job-labels: independent backlog item (general server-side labels)
- bug-1970: note the dropped REST 'tags' field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1 profile attributes: mode (direct), launch type, env + pre/post-exec,
essential resreq and limits, plus the job starter
- principle: a profile attribute is added by the spec that builds its job
feature; requeue/success exit values -> job-requeue (new backlog item),
checkpoint and warning-before-kill later, output path -> I/O track,
runtime estimate is not a profile attribute
- log bug-1970: REST submit accepts rerunable, requeue_error_values,
run_time_estimation, mig_threshold, chkpnt_*, warning_* and drops them
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- server presets live in <TopDir>/config/presets/ (or presets.dir in
app.yaml), one file per preset, re-read on change; not in ClusterConfig
- pull refuses a name collision unless it updates the same server preset;
--as / --force (keeps .bak); submit uses local presets only
- fix route prefix: /xce/v1/presets, not /api/v1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile gone or changed between scheduler decision and controller
dispatch: dispatch fails (retryable) and the job returns to PEND via
requeue, never EXIT
- internal per-profile content hash (canonical capnp, excluding
description) carried on the dispatch decision and the dispatch record
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles hold only what every job of the application must have; every
profile value binds and a job may add to it, never change or remove it
(replaces the default/fixed markers of 2026-10-09)
- convenience defaults move to client-side presets (new xcecli-presets
brief): option-level expansion on the client, the apiserver only shares
admin-deployed presets read-only, pull copies, export/import
- service section stays in the profile: probes, restart policy, ports bind;
replica count within the profile range; spread/rollout if unset
- job-config-layering, job-config-merge brief/requirements, roadmap and
resource-questions updated to the additive rule
- app:override and profile inheritance deferred; recommendations target
site presets
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles exist mainly so users need not repeat resreq and other settings;
most values are defaults, only what the application needs is fixed
- usage-based recommendations are advisory (user decision); new backlog
item profile-usage-insight
- open questions: per-term resreq markers with ranges, app:override,
inheritance, presets, job labels, compound resreqs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles in src/configs/applications.yaml (+ schema, ClusterConfig and
Conf4Sched), optional file with a built-in 'default' profile
- default_application in xce.yaml; an undefined name is a config error at
xceadmin build, live upgrade and controller start, never a fallback
- remove the superseded submit-time resolution text: the job keeps its
profile name and own values, effective values come from job-config-merge
- this spec owns the profile attribute schema and override markers
- record the 2026-10-12 open questions with leanings
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LLM training is one parallel job with topology requirements (PR-10).
gang-groups moves from the spec list to the roadmap backlog, re-scoped as
a co-start group of separately managed jobs and services (RL
post-training, disaggregated serving, streaming pipelines). resource-model
phases and the brief updated; job-resize no longer resizes gang groups.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Persistent log of the resource-design discussion: the user's proposals
(generic supply map incl. provisioned supply; a resource provider proxy
between the controller and upstream providers; datasets in the map with
staging as data provisioning) with analysis, the user's 2026-10-11 input
(duration/decay must-have but future; RBAC in the apiserver, discuss later;
OCI containers via application profiles; CSI compatibility preferred;
GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items
and new questions N1-N4, C3. resource-model.md, the provisioner research,
roadmap and CLAUDE.md point at it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
resource-model.md: one resource type descriptor and provider contract over
three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler
only selects from its cached snapshot, providers run after dispatch; batch
stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims
as WAL entities with the resource-provisioner discipline; LSF/K8s mapping;
LLM training/serving needs; phases; nine pending user decisions.
New briefs: custom-resource-reporting, resource-instances,
resource-claims, gang-groups. Roadmap: long-term LLM training + serving
platform direction (features from LSF and Kubernetes), the four new specs,
updates to ten existing specs, discovery note. resource-distribution-mapping
becomes phase 1 (shared pools + descriptor fields).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scaler is multi-threaded (monitor and demand threads); invoke() forked
and, in the child, logged, snprintf'd and setenv()'d before execl(), which
can deadlock on a lock another thread held at fork time and block the
caller in waitpid() forever.
run_script() now posix_spawn()s `sh <script> -f <in.json>`: stdout and
stderr go to the output file through file actions, the provider variables
(PRO_CONF_DIR, PRO_DATA_DIR, PRO_LOG_DIR, PROVIDER_NAME) are passed as the
child's envp (provider_env()), the child starts with an empty signal mask
and default dispositions, and waitpid() retries on EINTR.
The scaler unit suite (scaler/testsuite) was never built; it is now the
meson test "autoscaler test", linked with -Wl,--wrap=fork so it counts
fork() calls during a provider call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
RestartJobByID rebuilt the new job from the old job's row:
- it accepted asked_hosts and copied asked hosts, I/O paths and out_dir,
which no job gets (refused at submit since bug-1936); restart now
refuses asked_hosts with 400 and no longer carries those over;
- it read the JSONB columns chkpnt_info/extend_data as strings, but rows
keep models.Jsonb, so env/limits were lost and a checkpointed job never
restarted from its checkpoint; new rowJSON() reads both;
- it read "res_req" (the column is "resreq"), losing the resource
requirement, and never copied exec_home.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Design constraint (tech.md "Process Creation"): the controller, scheduler,
xcecd and anything that starts libxce threads never call fork, vfork,
forkpty, daemon, system, popen or uv_spawn; children start with posix_spawn()
or a systemd unit. A process proven single-threaded may fork, marked
/* xce-fork-ok: <why> */. Tests, harness, autoscaler (not finalized; bug-1969)
and the Go frontend are out of scope.
- scripts/check-no-fork.py scans src/{controller,scheduler,lib,worker}
(comment/string/#if 0 aware); runs as meson test "no-fork rule" in the
lib/controller/scheduler/worker suites and in the pre-commit hook.
- worker: drop xcecd's unused fork-based worker start paths
(process_manager_start_process, process_manager_start_worker; xcecd
starts xcecd_worker over D-Bus) and a dead #if 0 daemonize fork.
- xcecd_worker's fork/uv_spawn/system calls are marked: its call graph
from main(), callbacks included, reaches no pthread_create or libuv
thread-pool call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: CreateJob accepted in_file, out_file, err_file, out_dir,
asked_hosts, begin_time, term_time and transfer_files and returned
success, but neither the controller's submit data/start info nor the
worker carries any of them (the worker never stages transferred files),
so they were silently dropped. Exec home had no submit field at all, and
a relative cwd would have been resolved against the worker unit's
directory.
Fix: new preCheckSubmitOptions() runs right after binding, before file
uploads and ID allocation, and answers 400 naming every unsupported
option that is set; cwd and the new exec_home field must be absolute and
fit the worker's job_assignment_t buffers (1023 bytes). exec_home goes
to JobInfo.ExecHome and from there (submitStartInfo) to the start info's
exechome. num_files already reaches exdata as nofile_limit; a handler
test now pins it. The array %J/%I submit-time substitution test (9.4)
checks cwd instead of out_file/err_file, which are now refused. Swagger
regenerated with `swag init -g cmd/main.go -o docs`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the start info's execcwd/exechome reached xcecd and its worker-list
entry, but the MSG_JOB_ASSIGNED reply to the per-job xcecd_worker
(job_assignment_t) carried neither, and uv_proc_exec() spawned the job
with cwd NULL, so every job ran in the xcecd_worker unit's directory
whatever --cwd said, and exec home was never applied. parse_limits()
ignored the apiserver's "nofile_limit", so num_files never became an
rlimit.
Fix: job_assignment_t gets cwd/home, filled by the new
ipc_fill_job_assignment() (uv_ipc.c, which also NUL-ends every copied
string). xcecd_worker's job callback calls the new
uv_proc_enter_job_dirs() (uv_proc.c) before pre-exec and the job: it
exports HOME and chdir()s into cwd, falling back to /tmp with a warning
when the directory cannot be entered (the xcecd_worker runs as the job
user, so this is the user's own access). parse_limits() reads
nofile_limit into res_limits_t.nofile_limit and start_systemd_job() sets
LimitNOFILE like LimitNPROC. xcecd_virtual spawns no job and is
unaffected.
Tests: testsuite/test_job_dirs.c (assignment carries cwd/home, enter
cwd/home, fallback, a spawned job's pwd is the cwd) and
test_apiserver_nofile_limit_parsed in test_exdata_parse.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller sends job control to every host of a multi-host job
(bug-1942) as soon as the job is dispatched, but a remote host's runtasks
request is relayed by the controller (task.c do_runtasks, stateless) only
after the head worker has started and asked for it. A kill/stop that
reached a remote host first found no worker and no pending start, was
dropped ("Failed to find worker for job ID"), and the task started later
and kept running.
pending_signal now also holds "early" entries: handle_jobcontrol() records
a signal for a job with no task here (pending_signal_record_early), and
handle_runtasks() adopts it (pending_signal_start_task), so it is delivered
at RUNNING through the bug-1908 path and the task's exit is reported as
usual. Early entries are bounded (256) and expire after 60 s, pruned on
every early record and task start. A late control for a task that already
finished here (finished-job retention) is still dropped, and a head-host
start does not adopt an early entry, so neither can hit a later run of the
same jobid.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
limit_seed_config_impl() key_add()ed each pushed limit into the existing
table and returned at the first failure, so on any second push (live
config upgrade, or a simulator CONFIG_PUSH since bug-1964) the first
already-present name failed with -EEXIST: changed limits kept the old
definition and later new limits were never added.
A push now builds the whole new limit set into a fresh table and swaps it
in (all-or-nothing: on any build/register failure the previous set and its
instances stay). Usage instances are carried across: an instance of a kept
limit whose scoping and resource are unchanged is rebound to the new
definition; an instance of a removed limit, or of one whose per-consumer
scoping, resource or percent changed, is dropped and re-created on the next
match. Usage itself lives in the resource manager under the instance's
virtual resource, so counts survive either way. Not part of the snapshot
(a snapshot carries the config and reseeds from it).
The seed moves to plugins/limit_config.c (limit.c is near the 800-line
cap) with the private data type in plugins/limit_internal.h. Two REPL
limit-table fixtures relied on the additive behaviour (two single-limit
pushes) and now configure both limits in one push.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
