Revocation and expiry close live connections authenticated with the key,
since authentication happens only at join. Lists the design questions:
where the trusted list is changed, keeping revoked keys revoked, bootstrap
keys, token registration for elastic hosts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Per-host identity without a CA; only public keys move, and the trusted
list can ride on the replicated cluster config. Recommended over the
phased per-role-keys/PKI plan; decision still deferred.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Promotes the roadmap item to a brief: production/test mode on the
controller, authenticated roles at join, per-role request allowlist with
user requests only from the apiserver, an admin role for xceadmin. Lays
out the open decisions on secret type (cluster key / per-role keys / PKI)
and key distribution (files / token enrolment), with a phased
recommendation. project-config Req 3.13 applies to test mode only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Production submits always go through the apiserver; harness and direct
client-library submits are test tools. The apiserver resolves the project
and checks project:submit; the controller records the project and the
authorization basis it is given. Direct submits are recorded as given
with basis `direct`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Submitting under a project becomes the RBAC permission project:submit,
implied by membership (that project only) and grantable by roles (scope
covers the subtree). A project's admins get every job permission and
project:view over its subtree; they cannot edit the project, which only
the cluster admin changes through config in this stage.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The project tree is the only place concurrent use is capped: quotas are
written on the project (quota, member_quota over the subtree) and compiled
into the limit plugin, which becomes internal. Queues keep per-job
ceilings only; host slot caps are supply. Project path is the identity
and the only submit form; members are users and user groups, never
sub-projects. Adds the limits-to-owners backlog item.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Preset routes need no RBAC permission; a submit made from a preset is
checked like any other submit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1: any queue:submit user may use any profile; listing reuses cluster:view
- app:use / app:view design kept in the brief and moved to the new backlog
item app-access-control (depends on rbac-persistence for API grants)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- worker-run hooks: pre/post_job_exec (once, first host) and
pre/post_host_exec (once per host); no per-task hooks
- task starter reaches MPI ranks through the mpirun line XCE builds
(correction); XCE_JOB_HOST_DIR, XCE_HOST_NTASKS, XCE_HOST_TASK_INDEX
- hook failure: job EXIT with a reason naming the hook; post hooks still run
- post hooks must survive every way a job ends: run limit is systemd
RuntimeMaxUSec stopping the unit (worker included) and post-exec is only
bounded by the unset TimeoutStopUSec; requirement for an explicit stop
timeout and OOM survival (not logged as a bug: post-exec does run today)
- default profile always usable with queue:submit; service profile
rejected for batch jobs; warning when no host matches the profile select
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- params: declared application settings XCE carries but never interprets;
exported as XCE_APP_PARAM_*; job overrides with --app-param; undeclared
names rejected; stored in exdata app_params
- annotations: metadata for tools, also exported as XCE_APP_ANN_* (name
conversion, collision check, size limits, no secrets)
- merge drops 0-valued rusage terms so the scheduler sees no term (today a
0 term still rejects hosts where the resource is over-committed)
- app-submit-hook (esub-style) recorded as a roadmap backlog item only
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gaps closed:
- profile env travels separately from the job env (needed for the clean
outer-layer environment; remote tasks get both)
- merge library parses and rewrites exdata (env, limits; keeps meta);
added to job-config-merge scope
- new start_info fields listed (job starter, profile env)
- empty profile name = no profile, never resolved to default_application
- a file profile named default replaces the built-in one
- restart copies the profile name; array jobs carry one name
- multi-host outer pre/post-exec and job starter placement as open design
questions; rusage=0 and declined-dispatch counting to confirm
- scheduler-side field completeness (frozen dispatched jobs, Conf4Sched
profiles in snapshot/replay)
- no secrets in profiles; xceadmin warns on secret-looking env names
Stale text fixed: job-modify reference, exdata wording, rusage defaults
exception, I/O directories (moved to job-sandbox-execution), sandbox in the
sketch, job starter and job_pre_post_exec in the table, dispatch record
contents, misplaced rationale paragraph, versioning reference, scope and
touchpoints, task range (min/max) in job-config-layering.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile resreq split into select / rusage entries / span / same / order;
rusage entries carry optional per: (default task, today's PER_DEFAULT
behaviour) and optional default
- job's rusage term wins whole for a resource in both; unit must match;
no modifier inheritance
- compound/alternative: only the profile select applies; any profile
rusage default or span/same/order rejects until PR-15
- job-config-layering: merge rules at a glance table; profile = essentials
plus default reservations
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- rusage: union of profile and job resources; for a resource in both the
job's value wins (either direction); the profile's per-type applies and an
explicitly different per-type is rejected; omitted modifiers inherited;
name=0 means none
- rusage is the one profile section that is a per-resource default;
shared-resource protection via limits.yaml/quotas, later a counted marker
- side effects recorded for requirements (lowering warnings, redundant
preset values, origin labels, ceiling errors, hash churn)
- compound/alternative: profile select ANDed into every part, per-task
rusage merged per part, other profile resreq sections reject until PR-15
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- no preset field on the server: the client puts preset name + hash into
the job's exdata JSON under 'meta', passed through uninterpreted; the
worker skips 'meta' silently; 'preset report' filters jobs client-side
- drop the local submission log idea
- job-labels: independent backlog item (general server-side labels)
- bug-1970: note the dropped REST 'tags' field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1 profile attributes: mode (direct), launch type, env + pre/post-exec,
essential resreq and limits, plus the job starter
- principle: a profile attribute is added by the spec that builds its job
feature; requeue/success exit values -> job-requeue (new backlog item),
checkpoint and warning-before-kill later, output path -> I/O track,
runtime estimate is not a profile attribute
- log bug-1970: REST submit accepts rerunable, requeue_error_values,
run_time_estimation, mig_threshold, chkpnt_*, warning_* and drops them
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- server presets live in <TopDir>/config/presets/ (or presets.dir in
app.yaml), one file per preset, re-read on change; not in ClusterConfig
- pull refuses a name collision unless it updates the same server preset;
--as / --force (keeps .bak); submit uses local presets only
- fix route prefix: /xce/v1/presets, not /api/v1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile gone or changed between scheduler decision and controller
dispatch: dispatch fails (retryable) and the job returns to PEND via
requeue, never EXIT
- internal per-profile content hash (canonical capnp, excluding
description) carried on the dispatch decision and the dispatch record
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles hold only what every job of the application must have; every
profile value binds and a job may add to it, never change or remove it
(replaces the default/fixed markers of 2026-10-09)
- convenience defaults move to client-side presets (new xcecli-presets
brief): option-level expansion on the client, the apiserver only shares
admin-deployed presets read-only, pull copies, export/import
- service section stays in the profile: probes, restart policy, ports bind;
replica count within the profile range; spread/rollout if unset
- job-config-layering, job-config-merge brief/requirements, roadmap and
resource-questions updated to the additive rule
- app:override and profile inheritance deferred; recommendations target
site presets
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles exist mainly so users need not repeat resreq and other settings;
most values are defaults, only what the application needs is fixed
- usage-based recommendations are advisory (user decision); new backlog
item profile-usage-insight
- open questions: per-term resreq markers with ranges, app:override,
inheritance, presets, job labels, compound resreqs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles in src/configs/applications.yaml (+ schema, ClusterConfig and
Conf4Sched), optional file with a built-in 'default' profile
- default_application in xce.yaml; an undefined name is a config error at
xceadmin build, live upgrade and controller start, never a fallback
- remove the superseded submit-time resolution text: the job keeps its
profile name and own values, effective values come from job-config-merge
- this spec owns the profile attribute schema and override markers
- record the 2026-10-12 open questions with leanings
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LLM training is one parallel job with topology requirements (PR-10).
gang-groups moves from the spec list to the roadmap backlog, re-scoped as
a co-start group of separately managed jobs and services (RL
post-training, disaggregated serving, streaming pipelines). resource-model
phases and the brief updated; job-resize no longer resizes gang groups.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Persistent log of the resource-design discussion: the user's proposals
(generic supply map incl. provisioned supply; a resource provider proxy
between the controller and upstream providers; datasets in the map with
staging as data provisioning) with analysis, the user's 2026-10-11 input
(duration/decay must-have but future; RBAC in the apiserver, discuss later;
OCI containers via application profiles; CSI compatibility preferred;
GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items
and new questions N1-N4, C3. resource-model.md, the provisioner research,
roadmap and CLAUDE.md point at it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
resource-model.md: one resource type descriptor and provider contract over
three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler
only selects from its cached snapshot, providers run after dispatch; batch
stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims
as WAL entities with the resource-provisioner discipline; LSF/K8s mapping;
LLM training/serving needs; phases; nine pending user decisions.
New briefs: custom-resource-reporting, resource-instances,
resource-claims, gang-groups. Roadmap: long-term LLM training + serving
platform direction (features from LSF and Kubernetes), the four new specs,
updates to ten existing specs, discovery note. resource-distribution-mapping
becomes phase 1 (shared pools + descriptor fields).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
