test_sim_reconcile_unaccounted.c captures stdout through mkstemp() files in
/tmp/xce_simreconcile_*, but only the shared capture_stdout_around() helper
unlinked its file after reading it back -- the 4 inlined capture sites left
theirs behind on every passing run (the mkdtemp shim does not intercept
mkstemp, so nothing else removed them). Add the same close(fd); unlink(tmp)
the file family's other capture helpers use to each of the 4 sites.
Verified: scheduler sim reconcile unaccounted test passes and leaves no
/tmp/xce_simreconcile_* files behind.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).
Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.
Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
queue refuses new submissions only; pending jobs still run) --
supersedes our QUEUE_CLOSED preason on the dispatch path;
sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
rounds (sched_stash_config/sched_apply_pending_config), applied at
the same main-thread round boundary that preserves bug-1513
serialization
- bug-1760: sched_check_pending_rejoin exposed +
xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
(superset of our bug-1500 execuser fallback; our job-typed
resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
src/harness/test.c; controller suite split into
controller_test.c + fixtures; their clitest fixtures spawn xce_test
mid-suite under `meson test -j`
Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
gate re-ported into their relocated allocator (sched_election.c)
after plugin_job_election, with attach on the allocator thread
only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
merge initially re-carried our first-detail-add-promotes-main_reason
fallback in add_job_preason (4e2ac86), but their whole trace/verify
stack treats "detail-only => main_reason 0" as a contract --
sched_trace_capture counts main_reason as its own occurrence
("Expected 2 Was 3" in test_trace_capture), the outcome renderer
prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
and the committed .replay/.sim_verify baselines encode main_reason=0
(test_repl_trace_baseline byte compare, test_trace_equivalence
off-vs-traced). The fallback is dropped from the shared lib;
the user-visible "overall reason" survives as an API-layer view:
rawJobToModel (convert_pure.go) promotes the first detail row to
main_reason_name when main_reason is 0 (main_reason VALUE marshals
unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
(their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
resend, xcecli_force_reconnect + their is_connection_broken both
exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
/xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)
bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
(testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
harness_test.c -- hundreds of cases per process) call
case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
dirs are swept immediately in tearDown (after engine reaping), FAILED
cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
disables. One full controller_unit_test run was observed to leak 12G+
to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
failures
- harness_unit_test_deps links the shim library as well
Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
detail-keeps-main-0 contract; new tests cover limit-code detail
rows and explicit-main-wins-over-later-details; apiserver
convert_test.go gains the detail-only name-promotion cases
Co-Authored-By: Claude Code <noreply@anthropic.com>
A job without a project and without a group default goes to the cluster
default project, the built-in `default` (members all) unless configured
otherwise; it is never rejected for lacking a project.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A share may be owned at a location (host group) as well as anywhere.
Per resource, located host groups and the rest of the cluster form
disjoint pools; usage is charged to the host's pool and every pool follows
the share rules on its own, so located ownership is real. Pool capacity
counts registered hosts including down and closed ones. No-default
submits list the user's projects; a per-user default project is a
client-side xcecli setting; members see their ancestors' usage, not
siblings' details.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- `jobs` is a built-in resource for shares and member quotas: the job
limits per project and user; pending caps go to pending-job-limits.
- Jobs may be charged to inner projects, with a warning and pend reason
when their unassigned part is empty.
- Project aliases keep jobs through renames; running jobs follow them.
- Changing a pending job's project is part of job-modify.
- New project-accounting brief: durable finished-job records, outside the
WAL, before controller-job-cleanup drops a job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fairshare weights and usage history live on the project tree; queues keep
priority between queues and only choose fairshare or FCFS ordering. The
per-queue `fairshare:` user lists in queues.yaml go away.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A project's `share` (absolute or a percentage of its parent's) is carved
from the parent; siblings never add up to more than the parent. v1 is
strict: a project holds at most its share. A project's own jobs and its
children without a share draw from its unassigned part. Borrowing idle
shares above a project's own (`max`) and reclaim are future work, in the
renamed project-borrowing brief.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Revocation and expiry close live connections authenticated with the key,
since authentication happens only at join. Lists the design questions:
where the trusted list is changed, keeping revoked keys revoked, bootstrap
keys, token registration for elastic hosts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Per-host identity without a CA; only public keys move, and the trusted
list can ride on the replicated cluster config. Recommended over the
phased per-role-keys/PKI plan; decision still deferred.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Promotes the roadmap item to a brief: production/test mode on the
controller, authenticated roles at join, per-role request allowlist with
user requests only from the apiserver, an admin role for xceadmin. Lays
out the open decisions on secret type (cluster key / per-role keys / PKI)
and key distribution (files / token enrolment), with a phased
recommendation. project-config Req 3.13 applies to test mode only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Production submits always go through the apiserver; harness and direct
client-library submits are test tools. The apiserver resolves the project
and checks project:submit; the controller records the project and the
authorization basis it is given. Direct submits are recorded as given
with basis `direct`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Submitting under a project becomes the RBAC permission project:submit,
implied by membership (that project only) and grantable by roles (scope
covers the subtree). A project's admins get every job permission and
project:view over its subtree; they cannot edit the project, which only
the cluster admin changes through config in this stage.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The project tree is the only place concurrent use is capped: quotas are
written on the project (quota, member_quota over the subtree) and compiled
into the limit plugin, which becomes internal. Queues keep per-job
ceilings only; host slot caps are supply. Project path is the identity
and the only submit form; members are users and user groups, never
sub-projects. Adds the limits-to-owners backlog item.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Preset routes need no RBAC permission; a submit made from a preset is
checked like any other submit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1: any queue:submit user may use any profile; listing reuses cluster:view
- app:use / app:view design kept in the brief and moved to the new backlog
item app-access-control (depends on rbac-persistence for API grants)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- worker-run hooks: pre/post_job_exec (once, first host) and
pre/post_host_exec (once per host); no per-task hooks
- task starter reaches MPI ranks through the mpirun line XCE builds
(correction); XCE_JOB_HOST_DIR, XCE_HOST_NTASKS, XCE_HOST_TASK_INDEX
- hook failure: job EXIT with a reason naming the hook; post hooks still run
- post hooks must survive every way a job ends: run limit is systemd
RuntimeMaxUSec stopping the unit (worker included) and post-exec is only
bounded by the unset TimeoutStopUSec; requirement for an explicit stop
timeout and OOM survival (not logged as a bug: post-exec does run today)
- default profile always usable with queue:submit; service profile
rejected for batch jobs; warning when no host matches the profile select
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- params: declared application settings XCE carries but never interprets;
exported as XCE_APP_PARAM_*; job overrides with --app-param; undeclared
names rejected; stored in exdata app_params
- annotations: metadata for tools, also exported as XCE_APP_ANN_* (name
conversion, collision check, size limits, no secrets)
- merge drops 0-valued rusage terms so the scheduler sees no term (today a
0 term still rejects hosts where the resource is over-committed)
- app-submit-hook (esub-style) recorded as a roadmap backlog item only
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gaps closed:
- profile env travels separately from the job env (needed for the clean
outer-layer environment; remote tasks get both)
- merge library parses and rewrites exdata (env, limits; keeps meta);
added to job-config-merge scope
- new start_info fields listed (job starter, profile env)
- empty profile name = no profile, never resolved to default_application
- a file profile named default replaces the built-in one
- restart copies the profile name; array jobs carry one name
- multi-host outer pre/post-exec and job starter placement as open design
questions; rusage=0 and declined-dispatch counting to confirm
- scheduler-side field completeness (frozen dispatched jobs, Conf4Sched
profiles in snapshot/replay)
- no secrets in profiles; xceadmin warns on secret-looking env names
Stale text fixed: job-modify reference, exdata wording, rusage defaults
exception, I/O directories (moved to job-sandbox-execution), sandbox in the
sketch, job starter and job_pre_post_exec in the table, dispatch record
contents, misplaced rationale paragraph, versioning reference, scope and
touchpoints, task range (min/max) in job-config-layering.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile resreq split into select / rusage entries / span / same / order;
rusage entries carry optional per: (default task, today's PER_DEFAULT
behaviour) and optional default
- job's rusage term wins whole for a resource in both; unit must match;
no modifier inheritance
- compound/alternative: only the profile select applies; any profile
rusage default or span/same/order rejects until PR-15
- job-config-layering: merge rules at a glance table; profile = essentials
plus default reservations
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- rusage: union of profile and job resources; for a resource in both the
job's value wins (either direction); the profile's per-type applies and an
explicitly different per-type is rejected; omitted modifiers inherited;
name=0 means none
- rusage is the one profile section that is a per-resource default;
shared-resource protection via limits.yaml/quotas, later a counted marker
- side effects recorded for requirements (lowering warnings, redundant
preset values, origin labels, ceiling errors, hash churn)
- compound/alternative: profile select ANDed into every part, per-task
rusage merged per part, other profile resreq sections reject until PR-15
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- no preset field on the server: the client puts preset name + hash into
the job's exdata JSON under 'meta', passed through uninterpreted; the
worker skips 'meta' silently; 'preset report' filters jobs client-side
- drop the local submission log idea
- job-labels: independent backlog item (general server-side labels)
- bug-1970: note the dropped REST 'tags' field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- v1 profile attributes: mode (direct), launch type, env + pre/post-exec,
essential resreq and limits, plus the job starter
- principle: a profile attribute is added by the spec that builds its job
feature; requeue/success exit values -> job-requeue (new backlog item),
checkpoint and warning-before-kill later, output path -> I/O track,
runtime estimate is not a profile attribute
- log bug-1970: REST submit accepts rerunable, requeue_error_values,
run_time_estimation, mig_threshold, chkpnt_*, warning_* and drops them
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- server presets live in <TopDir>/config/presets/ (or presets.dir in
app.yaml), one file per preset, re-read on change; not in ClusterConfig
- pull refuses a name collision unless it updates the same server preset;
--as / --force (keeps .bak); submit uses local presets only
- fix route prefix: /xce/v1/presets, not /api/v1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profile gone or changed between scheduler decision and controller
dispatch: dispatch fails (retryable) and the job returns to PEND via
requeue, never EXIT
- internal per-profile content hash (canonical capnp, excluding
description) carried on the dispatch decision and the dispatch record
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles hold only what every job of the application must have; every
profile value binds and a job may add to it, never change or remove it
(replaces the default/fixed markers of 2026-10-09)
- convenience defaults move to client-side presets (new xcecli-presets
brief): option-level expansion on the client, the apiserver only shares
admin-deployed presets read-only, pull copies, export/import
- service section stays in the profile: probes, restart policy, ports bind;
replica count within the profile range; spread/rollout if unset
- job-config-layering, job-config-merge brief/requirements, roadmap and
resource-questions updated to the additive rule
- app:override and profile inheritance deferred; recommendations target
site presets
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles exist mainly so users need not repeat resreq and other settings;
most values are defaults, only what the application needs is fixed
- usage-based recommendations are advisory (user decision); new backlog
item profile-usage-insight
- open questions: per-term resreq markers with ranges, app:override,
inheritance, presets, job labels, compound resreqs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- profiles in src/configs/applications.yaml (+ schema, ClusterConfig and
Conf4Sched), optional file with a built-in 'default' profile
- default_application in xce.yaml; an undefined name is a config error at
xceadmin build, live upgrade and controller start, never a fallback
- remove the superseded submit-time resolution text: the job keeps its
profile name and own values, effective values come from job-config-merge
- this spec owns the profile attribute schema and override markers
- record the 2026-10-12 open questions with leanings
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LLM training is one parallel job with topology requirements (PR-10).
gang-groups moves from the spec list to the roadmap backlog, re-scoped as
a co-start group of separately managed jobs and services (RL
post-training, disaggregated serving, streaming pipelines). resource-model
phases and the brief updated; job-resize no longer resizes gang groups.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
