- worker-run hooks: pre/post_job_exec (once, first host) and
pre/post_host_exec (once per host); no per-task hooks
- task starter reaches MPI ranks through the mpirun line XCE builds
(correction); XCE_JOB_HOST_DIR, XCE_HOST_NTASKS, XCE_HOST_TASK_INDEX
- hook failure: job EXIT with a reason naming the hook; post hooks still run
- post hooks must survive every way a job ends: run limit is systemd
RuntimeMaxUSec stopping the unit (worker included) and post-exec is only
bounded by the unset TimeoutStopUSec; requirement for an explicit stop
timeout and OOM survival (not logged as a bug: post-exec does run today)
- default profile always usable with queue:submit; service profile
rejected for batch jobs; warning when no host matches the profile select
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- params: declared application settings XCE carries but never interprets;
exported as XCE_APP_PARAM_*; job overrides with --app-param; undeclared
names rejected; stored in exdata app_params
- annotations: metadata for tools, also exported as XCE_APP_ANN_* (name
conversion, collision check, size limits, no secrets)
- merge drops 0-valued rusage terms so the scheduler sees no term (today a
0 term still rejects hosts where the resource is over-committed)
- app-submit-hook (esub-style) recorded as a roadmap backlog item only
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gaps closed:
- profile env travels separately from the job env (needed for the clean
outer-layer environment; remote tasks get both)
- merge library parses and rewrites exdata (env, limits; keeps meta);
added to job-config-merge scope
- new start_info fields listed (job starter, profile env)
- empty profile name = no profile, never resolved to default_application
- a file profile named default replaces the built-in one
- restart copies the profile name; array jobs carry one name
- multi-host outer pre/post-exec and job starter placement as open design
questions; rusage=0 and declined-dispatch counting to confirm
- scheduler-side field completeness (frozen dispatched jobs, Conf4Sched
profiles in snapshot/replay)
- no secrets in profiles; xceadmin warns on secret-looking env names
Stale text fixed: job-modify reference, exdata wording, rusage defaults
exception, I/O directories (moved to job-sandbox-execution), sandbox in the
sketch, job starter and job_pre_post_exec in the table, dispatch record
contents, misplaced rationale paragraph, versioning reference, scope and
touchpoints, task range (min/max) in job-config-layering.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile resreq split into select / rusage entries / span / same / order;
rusage entries carry optional per: (default task, today's PER_DEFAULT
behaviour) and optional default
- job's rusage term wins whole for a resource in both; unit must match;
no modifier inheritance
- compound/alternative: only the profile select applies; any profile
rusage default or span/same/order rejects until PR-15
- job-config-layering: merge rules at a glance table; profile = essentials
plus default reservations
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- rusage: union of profile and job resources; for a resource in both the
job's value wins (either direction); the profile's per-type applies and an
explicitly different per-type is rejected; omitted modifiers inherited;
name=0 means none
- rusage is the one profile section that is a per-resource default;
shared-resource protection via limits.yaml/quotas, later a counted marker
- side effects recorded for requirements (lowering warnings, redundant
preset values, origin labels, ceiling errors, hash churn)
- compound/alternative: profile select ANDed into every part, per-task
rusage merged per part, other profile resreq sections reject until PR-15
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- no preset field on the server: the client puts preset name + hash into
the job's exdata JSON under 'meta', passed through uninterpreted; the
worker skips 'meta' silently; 'preset report' filters jobs client-side
- drop the local submission log idea
- job-labels: independent backlog item (general server-side labels)
- bug-1970: note the dropped REST 'tags' field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1 profile attributes: mode (direct), launch type, env + pre/post-exec,
essential resreq and limits, plus the job starter
- principle: a profile attribute is added by the spec that builds its job
feature; requeue/success exit values -> job-requeue (new backlog item),
checkpoint and warning-before-kill later, output path -> I/O track,
runtime estimate is not a profile attribute
- log bug-1970: REST submit accepts rerunable, requeue_error_values,
run_time_estimation, mig_threshold, chkpnt_*, warning_* and drops them
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- server presets live in <TopDir>/config/presets/ (or presets.dir in
app.yaml), one file per preset, re-read on change; not in ClusterConfig
- pull refuses a name collision unless it updates the same server preset;
--as / --force (keeps .bak); submit uses local presets only
- fix route prefix: /xce/v1/presets, not /api/v1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile gone or changed between scheduler decision and controller
dispatch: dispatch fails (retryable) and the job returns to PEND via
requeue, never EXIT
- internal per-profile content hash (canonical capnp, excluding
description) carried on the dispatch decision and the dispatch record
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles hold only what every job of the application must have; every
profile value binds and a job may add to it, never change or remove it
(replaces the default/fixed markers of 2026-10-09)
- convenience defaults move to client-side presets (new xcecli-presets
brief): option-level expansion on the client, the apiserver only shares
admin-deployed presets read-only, pull copies, export/import
- service section stays in the profile: probes, restart policy, ports bind;
replica count within the profile range; spread/rollout if unset
- job-config-layering, job-config-merge brief/requirements, roadmap and
resource-questions updated to the additive rule
- app:override and profile inheritance deferred; recommendations target
site presets
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles exist mainly so users need not repeat resreq and other settings;
most values are defaults, only what the application needs is fixed
- usage-based recommendations are advisory (user decision); new backlog
item profile-usage-insight
- open questions: per-term resreq markers with ranges, app:override,
inheritance, presets, job labels, compound resreqs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles in src/configs/applications.yaml (+ schema, ClusterConfig and
Conf4Sched), optional file with a built-in 'default' profile
- default_application in xce.yaml; an undefined name is a config error at
xceadmin build, live upgrade and controller start, never a fallback
- remove the superseded submit-time resolution text: the job keeps its
profile name and own values, effective values come from job-config-merge
- this spec owns the profile attribute schema and override markers
- record the 2026-10-12 open questions with leanings
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LLM training is one parallel job with topology requirements (PR-10).
gang-groups moves from the spec list to the roadmap backlog, re-scoped as
a co-start group of separately managed jobs and services (RL
post-training, disaggregated serving, streaming pipelines). resource-model
phases and the brief updated; job-resize no longer resizes gang groups.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Persistent log of the resource-design discussion: the user's proposals
(generic supply map incl. provisioned supply; a resource provider proxy
between the controller and upstream providers; datasets in the map with
staging as data provisioning) with analysis, the user's 2026-10-11 input
(duration/decay must-have but future; RBAC in the apiserver, discuss later;
OCI containers via application profiles; CSI compatibility preferred;
GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items
and new questions N1-N4, C3. resource-model.md, the provisioner research,
roadmap and CLAUDE.md point at it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
resource-model.md: one resource type descriptor and provider contract over
three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler
only selects from its cached snapshot, providers run after dispatch; batch
stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims
as WAL entities with the resource-provisioner discipline; LSF/K8s mapping;
LLM training/serving needs; phases; nine pending user decisions.
New briefs: custom-resource-reporting, resource-instances,
resource-claims, gang-groups. Roadmap: long-term LLM training + serving
platform direction (features from LSF and Kubernetes), the four new specs,
updates to ten existing specs, discovery note. resource-distribution-mapping
becomes phase 1 (shared pools + descriptor fields).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce-node-manager: a "Cross-Spec Interactions" section (pending a
requirements revision; approved criteria unchanged) with six points: a
supervision/probe module shared with service-supervision, node-manager
components vs service jobs, a lease-based provisioner role, role
declaration and worker stop on decommission for provisioned hosts, a
host-loss grace period longer than worker restart plus rejoin, and a future
service proxy under the reverse-proxy role. design.md Open Questions points
to it.
The service-supervision, host-failure-job-recovery, elastic-hosts,
resource-provisioner and service-endpoint briefs cross-reference the
matching points; the roadmap lists the xce-node-manager revision.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New briefs: provisioning-demand (the scheduler knows a provisionable
catalog, tries host templates as virtual hosts and license headroom after
real resources fail, finalizes demand and idle supply at session end;
simulator support is a hard requirement), resource-provisioner (new
daemon, several instances owning providers via controller leases,
self-describing providers, automatic policy, durable request/instance
state; src/autoscaler is prior art only), elastic-hosts (host templates,
expected hosts, drain and decommission), elastic-shared-resources
(borrowed license tokens change the shared total).
Roadmap: the new specs, plus updates to resource-distribution-mapping
(shared pools) and peer-authentication-hardening (provisioner role).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New briefs: host-failure-job-recovery (jobs on a lost host are requeued or
end EXIT after a grace period; stale copies fenced on rejoin; manual
requeue), service-supervision (profile kind service: restart policy and
liveness/readiness probes run by each replica's own worker, in-place
restart), service-replica-set (controller-held service object owning
independent replica jobs; event-driven replacement, scaling and rolling
update; no head agent, no promotion), service-endpoint (per-replica ports,
endpoints of ready replicas), job-resize (backlog, elastic jobs).
application-profile brief gains the service section; roadmap lists the new
specs, the existing-spec updates and the backlog item.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
job-config-merge: spec initialized from its discovery brief.
project-config: requirements generated (9 requirements): nested projects
named by path, membership-based submit check with no inheritance and
overlapping membership, group-default then cluster-default project, open
mode until projects are defined, pending jobs held with a reason after
project changes, roll-up quotas, project visibility, a project scope for
job permissions covering the subtree and a project:view permission,
compatibility.
Roadmap: project-delegated-membership backlog item (runtime membership
edits by project managers, deferred).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New steering job-config-layering.md: queue = scheduling policy, application
profile = what the application needs and how it runs (resreq, task range,
execution settings, pre/post-exec, limits), project = resource ownership.
Only two interactions: the job overrides a profile attribute marked
default (fixed: not at all; select: only ANDed terms), and queue ceilings
or project quotas check effective values (reject if never satisfiable,
pend if full now). Replaces the simplified job > app > queue rules.
Queue pre_exec/post_exec are deprecated with queue res_req: the profile's
pre/post-exec fills the worker's existing outer layer, the job's own stays
the inner layer, allowed or forbidden by the profile.
job-config-merge and application-profile briefs and the roadmap follow;
CLAUDE.md lists the new steering file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New briefs: job-config-merge (one live merge of job, application profile,
queue policy and project into effective values; simplified rules: job
overrides profile, select ANDed, queue hard/task limits reject; queue is
policy only, res_req deprecated), project-config (project quotas v1) and
project-guarantee (backlog).
application-profile: the profile carries the job's resreq (incl. span) and
task limits; "frozen at submit" replaced by the live merge.
parjob-placement: span constraints live in the fixed-N planner and the
range search bisects over the planner's allowed counts; LSF 10.1 block,
ptile, stripe and HOSTLIMIT_PER_JOB findings recorded.
Roadmap: new specs, existing-spec updates, lsf-config-migration backlog.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parjob-placement now covers full LSF-style parallel placement: task
range (-n min,max), capacity-driven partitioning, span[ptile=n],
span[hosts=1], span[block=n] and a host-count limit. bug-1931 points to
it for the full task-range behaviour.
pend-reason-push-dedup: skip pushing a job's pend reason to the
controller when it equals the last one the controller accepted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The session-level unsatisfiable-resreq cache stored only the exhausted
attempt's main reason (an int). The allocators record shortfalls as
per-host/per-resource entries and leave main_reason at 0, so every cache
hit in the live scheduler pended its job with an empty reason. The
simulator registered no cache at all (its table lived on xce_scheduler_t),
so it searched every identical pending job again and its traces differed
from production.
- The cache table moves to xce_sched_core_t; cache_lookup/insert/clear and
the hook adapters take the core, with no cast to a driver struct.
sched_init() builds it, cache_clear() rebuilds it per session,
xce_sched_core_fini() destroys it.
- A cache entry is a deep copy of the attempt's full xce_job_preason_t
plus the job that proved the request unsatisfiable (core->alloc_jobid,
set by sched_allocator()). A hit moves a copy into ctx->reason; a miss
leaves ctx->reason untouched.
- main.c passes the same three adapters to the simulator through new
sched_sim_args_t fields; sched_sim_driver_start() sets them on the sim
core (plain field writes: white-box tests include sim.c alone).
- Decision trace: a hit renders "no plan generated: skipped, same request
as <job> proved unsatisfiable earlier this session" (new
XCE_TRACE_NOPLAN_KNOWN_UNSAT via sched_trace_known_unsat()); the outcome
shows the restored reasons.
- sim_verify keeps its main-only PEND compare (old recordings carry
main-only hits); comments and the scheduler-sim-live-allocation design
note updated.
Tests: full-reason assertions in the cache dispatch tests (RED before:
"a cache hit must restore the original attempt's FULL pend reason"),
test_alloc_cache.c ported to the new API (deep copy, clear frees, -EEXIST
frees), a sim-registered clear test, a renderer test, and a sim e2e
scenario with two identical unsatisfiable jobs (204, 205) where 205 is
skipped with 204's full reasons (RED before: no skip line).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1931: the parjob allocator ignores minTasks (a tasks 2..5 job pends
although 2 slots are free).
- bug-1932: the unsatisfiable-request cache drops pend reasons on a hit
and is unused in the simulator.
- .kiro/specs/parjob-capacity-partitioning: initialized spec with the
candidate heuristics for capacity-driven plan generation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Feature validation round 1: tests for trace off keeping recall (1.5),
no logging from REPL trace sources (8.7), reconstructed records in the
trace file (8.1) and first in continue mode (7.3), output after trace
off matching an untraced run (10.4), and tracing enabled only from the
REPL (11.1). design.md now reflects the REPL-side file split, the added
API, the dropped file tag and the test files; duplicate trace
declarations removed from repl_internal.h.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 6.4: a 1000-host, 64-task parallel job traced at full level stores
exactly the line bound with exact omitted counts and counters; 10,000
traced decisions keep the trace store within 64 MiB without evicting the
newest. The scheduler guide lists the REPL trace files and test rules
and carries measured file sizes. Full all-components ASAN build passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 6.3: the contention and extra replays run with tracing off, all
jobs at summary and full level, and a job subset, in step and continue
mode; decisions, full pend reasons, --sim-output recordings, snapshots,
--sim-verify results and per-step accounting dumps are identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 6.2: injects plan-claims growth, snapshot, finalize, host-scan,
per-host uncommitted lookup, allocation-function and unknown-jobid
faults and asserts each traced outcome. Also fixes two deferred OOM
paths: a failed limit row keeps the host's verdict bits, and an array
listing that cannot be built is reported instead of silently truncated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 6.1: drives the real simulator REPL with tracing on for all and
selected jobs at both levels, asserting every Requirement 2-6 section,
ordering in step and continue mode and at breakpoints, info trace at a
breakpoint, after finish, for arrays and after a units change, and a
byte-identical trace file copy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 5.3: session hooks around each sim session; trace-only decisions
are handled first and skip verification, recording, echo and
breakpoints; traced decisions are emitted after the echo gate and before
the breakpoint check; reconstructed dispatches get reconstructed
records; unclaimed traces are freed at both decision free sites.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 5.2: trace is routed with the raw line and info trace through info;
help gains a Trace section; set output rejects open trace files; quit
and EOF shut tracing down; the continue-mode rollup reports a traced
round on its own; repl.h gains the interactive-only trace gates. The
baseline transcript changes only in the help and info-usage lines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 5.1: the 26 white-box tests that include repl.c or sim_loop.c now
include the REPL trace sources (and the core trace sources where no
hooked source brings them); sched_sim_isolation_lib carries all trace
sources and the sim isolation test scans the REPL trace files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 4.6: session begin makes the next settings current, stamps the
session number and publishes them to the core; session end withdraws and
releases them. Emission marks the round traced, flushes the collapsed
range and reprints the suppressed echo in continue mode, renders to the
console and the trace file with identical bytes, and stores the record.
sim_print_decision is exported unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 4.5: a named trace file is opened for appending at command time;
an open failure or a path that is the sim output file or the scheduler
log (same device and inode) is rejected with settings unchanged; files
close when their last settings holder releases them; set output gets a
conflict check; shutdown flushes and closes every trace file and frees
the store.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
