A share may be owned at a location (host group) as well as anywhere.
Per resource, located host groups and the rest of the cluster form
disjoint pools; usage is charged to the host's pool and every pool follows
the share rules on its own, so located ownership is real. Pool capacity
counts registered hosts including down and closed ones. No-default
submits list the user's projects; a per-user default project is a
client-side xcecli setting; members see their ancestors' usage, not
siblings' details.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- `jobs` is a built-in resource for shares and member quotas: the job
limits per project and user; pending caps go to pending-job-limits.
- Jobs may be charged to inner projects, with a warning and pend reason
when their unassigned part is empty.
- Project aliases keep jobs through renames; running jobs follow them.
- Changing a pending job's project is part of job-modify.
- New project-accounting brief: durable finished-job records, outside the
WAL, before controller-job-cleanup drops a job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fairshare weights and usage history live on the project tree; queues keep
priority between queues and only choose fairshare or FCFS ordering. The
per-queue `fairshare:` user lists in queues.yaml go away.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A project's `share` (absolute or a percentage of its parent's) is carved
from the parent; siblings never add up to more than the parent. v1 is
strict: a project holds at most its share. A project's own jobs and its
children without a share draw from its unassigned part. Borrowing idle
shares above a project's own (`max`) and reclaim are future work, in the
renamed project-borrowing brief.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Revocation and expiry close live connections authenticated with the key,
since authentication happens only at join. Lists the design questions:
where the trusted list is changed, keeping revoked keys revoked, bootstrap
keys, token registration for elastic hosts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Per-host identity without a CA; only public keys move, and the trusted
list can ride on the replicated cluster config. Recommended over the
phased per-role-keys/PKI plan; decision still deferred.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Promotes the roadmap item to a brief: production/test mode on the
controller, authenticated roles at join, per-role request allowlist with
user requests only from the apiserver, an admin role for xceadmin. Lays
out the open decisions on secret type (cluster key / per-role keys / PKI)
and key distribution (files / token enrolment), with a phased
recommendation. project-config Req 3.13 applies to test mode only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Production submits always go through the apiserver; harness and direct
client-library submits are test tools. The apiserver resolves the project
and checks project:submit; the controller records the project and the
authorization basis it is given. Direct submits are recorded as given
with basis `direct`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Submitting under a project becomes the RBAC permission project:submit,
implied by membership (that project only) and grantable by roles (scope
covers the subtree). A project's admins get every job permission and
project:view over its subtree; they cannot edit the project, which only
the cluster admin changes through config in this stage.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The project tree is the only place concurrent use is capped: quotas are
written on the project (quota, member_quota over the subtree) and compiled
into the limit plugin, which becomes internal. Queues keep per-job
ceilings only; host slot caps are supply. Project path is the identity
and the only submit form; members are users and user groups, never
sub-projects. Adds the limits-to-owners backlog item.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Preset routes need no RBAC permission; a submit made from a preset is
checked like any other submit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1: any queue:submit user may use any profile; listing reuses cluster:view
- app:use / app:view design kept in the brief and moved to the new backlog
item app-access-control (depends on rbac-persistence for API grants)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- worker-run hooks: pre/post_job_exec (once, first host) and
pre/post_host_exec (once per host); no per-task hooks
- task starter reaches MPI ranks through the mpirun line XCE builds
(correction); XCE_JOB_HOST_DIR, XCE_HOST_NTASKS, XCE_HOST_TASK_INDEX
- hook failure: job EXIT with a reason naming the hook; post hooks still run
- post hooks must survive every way a job ends: run limit is systemd
RuntimeMaxUSec stopping the unit (worker included) and post-exec is only
bounded by the unset TimeoutStopUSec; requirement for an explicit stop
timeout and OOM survival (not logged as a bug: post-exec does run today)
- default profile always usable with queue:submit; service profile
rejected for batch jobs; warning when no host matches the profile select
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- params: declared application settings XCE carries but never interprets;
exported as XCE_APP_PARAM_*; job overrides with --app-param; undeclared
names rejected; stored in exdata app_params
- annotations: metadata for tools, also exported as XCE_APP_ANN_* (name
conversion, collision check, size limits, no secrets)
- merge drops 0-valued rusage terms so the scheduler sees no term (today a
0 term still rejects hosts where the resource is over-committed)
- app-submit-hook (esub-style) recorded as a roadmap backlog item only
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gaps closed:
- profile env travels separately from the job env (needed for the clean
outer-layer environment; remote tasks get both)
- merge library parses and rewrites exdata (env, limits; keeps meta);
added to job-config-merge scope
- new start_info fields listed (job starter, profile env)
- empty profile name = no profile, never resolved to default_application
- a file profile named default replaces the built-in one
- restart copies the profile name; array jobs carry one name
- multi-host outer pre/post-exec and job starter placement as open design
questions; rusage=0 and declined-dispatch counting to confirm
- scheduler-side field completeness (frozen dispatched jobs, Conf4Sched
profiles in snapshot/replay)
- no secrets in profiles; xceadmin warns on secret-looking env names
Stale text fixed: job-modify reference, exdata wording, rusage defaults
exception, I/O directories (moved to job-sandbox-execution), sandbox in the
sketch, job starter and job_pre_post_exec in the table, dispatch record
contents, misplaced rationale paragraph, versioning reference, scope and
touchpoints, task range (min/max) in job-config-layering.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile resreq split into select / rusage entries / span / same / order;
rusage entries carry optional per: (default task, today's PER_DEFAULT
behaviour) and optional default
- job's rusage term wins whole for a resource in both; unit must match;
no modifier inheritance
- compound/alternative: only the profile select applies; any profile
rusage default or span/same/order rejects until PR-15
- job-config-layering: merge rules at a glance table; profile = essentials
plus default reservations
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- rusage: union of profile and job resources; for a resource in both the
job's value wins (either direction); the profile's per-type applies and an
explicitly different per-type is rejected; omitted modifiers inherited;
name=0 means none
- rusage is the one profile section that is a per-resource default;
shared-resource protection via limits.yaml/quotas, later a counted marker
- side effects recorded for requirements (lowering warnings, redundant
preset values, origin labels, ceiling errors, hash churn)
- compound/alternative: profile select ANDed into every part, per-task
rusage merged per part, other profile resreq sections reject until PR-15
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- no preset field on the server: the client puts preset name + hash into
the job's exdata JSON under 'meta', passed through uninterpreted; the
worker skips 'meta' silently; 'preset report' filters jobs client-side
- drop the local submission log idea
- job-labels: independent backlog item (general server-side labels)
- bug-1970: note the dropped REST 'tags' field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1 profile attributes: mode (direct), launch type, env + pre/post-exec,
essential resreq and limits, plus the job starter
- principle: a profile attribute is added by the spec that builds its job
feature; requeue/success exit values -> job-requeue (new backlog item),
checkpoint and warning-before-kill later, output path -> I/O track,
runtime estimate is not a profile attribute
- log bug-1970: REST submit accepts rerunable, requeue_error_values,
run_time_estimation, mig_threshold, chkpnt_*, warning_* and drops them
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- server presets live in <TopDir>/config/presets/ (or presets.dir in
app.yaml), one file per preset, re-read on change; not in ClusterConfig
- pull refuses a name collision unless it updates the same server preset;
--as / --force (keeps .bak); submit uses local presets only
- fix route prefix: /xce/v1/presets, not /api/v1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile gone or changed between scheduler decision and controller
dispatch: dispatch fails (retryable) and the job returns to PEND via
requeue, never EXIT
- internal per-profile content hash (canonical capnp, excluding
description) carried on the dispatch decision and the dispatch record
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles hold only what every job of the application must have; every
profile value binds and a job may add to it, never change or remove it
(replaces the default/fixed markers of 2026-10-09)
- convenience defaults move to client-side presets (new xcecli-presets
brief): option-level expansion on the client, the apiserver only shares
admin-deployed presets read-only, pull copies, export/import
- service section stays in the profile: probes, restart policy, ports bind;
replica count within the profile range; spread/rollout if unset
- job-config-layering, job-config-merge brief/requirements, roadmap and
resource-questions updated to the additive rule
- app:override and profile inheritance deferred; recommendations target
site presets
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles exist mainly so users need not repeat resreq and other settings;
most values are defaults, only what the application needs is fixed
- usage-based recommendations are advisory (user decision); new backlog
item profile-usage-insight
- open questions: per-term resreq markers with ranges, app:override,
inheritance, presets, job labels, compound resreqs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles in src/configs/applications.yaml (+ schema, ClusterConfig and
Conf4Sched), optional file with a built-in 'default' profile
- default_application in xce.yaml; an undefined name is a config error at
xceadmin build, live upgrade and controller start, never a fallback
- remove the superseded submit-time resolution text: the job keeps its
profile name and own values, effective values come from job-config-merge
- this spec owns the profile attribute schema and override markers
- record the 2026-10-12 open questions with leanings
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LLM training is one parallel job with topology requirements (PR-10).
gang-groups moves from the spec list to the roadmap backlog, re-scoped as
a co-start group of separately managed jobs and services (RL
post-training, disaggregated serving, streaming pipelines). resource-model
phases and the brief updated; job-resize no longer resizes gang groups.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
