Cause: preCheckJobRequest (controllers/jobs.go) ran the reserved-name,
existence and closed checks only when a queue name was given; an empty
name was set to "normal" and skipped them, so jobs were accepted into a
closed default queue.
Fix: default the name first, then run the same checks for every
submission. Two existing tests now seed the default queue. (The controller
side of bug-1960 is fixed separately.)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: isExcludedPath (middleware/validatehandler.go) dropped the last
character of every excluded entry and prefix-matched, so /ping excused any
path starting with /pin (e.g. /pinX) from JWT validation, while
/swagger/*any (trimmed to /swagger/*an) never matched.
Fix: entries match exactly; an entry whose last segment is a gin catch-all
(/swagger/*any) matches any path under the part before the "*".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: validateHost (controllers/hosts.go) wrote a 403 via ReturnError
without aborting, so ControlHost's ctx.IsAborted() check never fired and it
went on to write a second (202) response; the permission was also checked
once per host and not at all for an empty host list.
Fix: validateHost writes no response; it checks host:control once up front
and returns an error, on which ControlHost writes a single 403 and returns.
For "all", the host list is resolved before the one validateHost call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: RoleController.CreateRole, AddPermissionsForRole and
DeleteRoleAndPermissions relied on RequirePermission("rbac:control"), which
accepts any scope, so a user with rbac:control scoped to self could add
cluster:control to a role it holds.
Fix: checkRoleChangePermission() in controllers/roles.go refuses with 403
unless one of the caller's grants (checked before deduplication) is
rbac:control with scope all, reusing canChangeUserRoles from bug-1952.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control API already takes `signal` (kill only: that signal once,
instead of the graceful INT, TERM, KILL sequence) and `force_kill`
(SIGKILL at once); xcecli could not send either. `jobs kill` now has
-s/--signal <name|number> (names in any case, SIG prefix optional, sent
upper case without it) and --force, in both the job-ID and the filter
request; using both at once, or an empty signal, is refused before any
request. stop and resume get neither (the server rejects a signal there).
Kill help and docs/xce.md "Killing and stopping jobs" document the flags.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli logout only removed the token from the local config, so the token
stayed valid on the server until it expired. It now first sends
POST /xce/v1/auth/logout with the bearer token (APIClient.Logout), which
revokes it server side since bug-1946. Best effort: if the server is
unreachable or rejects the token, the local login info is still cleared
and a warning says the revocation could not be confirmed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- src/docs/xce.md: explicit signal and force kill are REST-only (signal,
force_kill); xcecli has no signal option yet.
- src/docs/deployment.md: app.yaml row lists the job block
(max_array_size, terminate_interval) and trusted proxies.
- xcecli jobs kill/stop/resume help describes the signal sequences.
- REST: control endpoint annotations and the signal/force_kill model fields
describe the behaviour (swagger output not regenerated: swag fails on an
unrelated annotation).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The loader now stores the effective interval (10 s when unset or 0, LSF's
JOB_TERMINATE_INTERVAL default) and refuses a negative one; app_schema.json
describes the setting; src/docs/xce.md documents kill/stop behaviour and
the interval.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A plain kill sent one SIGTERM and never escalated, and a plain stop sent
one uncatchable SIGSTOP. Following LSF's TERMINATE and SUSPEND actions, a
plain kill now sends SIGINT, then SIGTERM, then SIGKILL, and a plain stop
sends SIGTSTP (the job can catch it, prepare and stop itself), then SIGSTOP
if it has not stopped; each next signal follows after the grace period and
only while the job is alive (or not yet stopped). A resume cancels a
pending stop; the job's exit ends any sequence. An explicit signal is
delivered once; a force kill is SIGKILL at once.
- JobSignal gains grace (capnp @2, xce_signal_t.grace): > 0 asks for the
sequence, in seconds between signals.
- apiserver: job.terminate_interval (default 10 s, LSF's
JOB_TERMINATE_INTERVAL); jobCtrlParams sends kill = SIGKILL and stop =
SIGSTOP with that grace when no signal is given; the handler no longer
pre-fills a default signal.
- worker: grace travels through the job-control task, the kept-signal table
(pending_signal_*_ex) and the IPC message; the job process runs the
sequence (job_signal_seq_*, uv timer) and checks stop state in /proc.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only a head-host job start opened the pending-signal window (bug-1908), so a
kill for a remote task of a multi-host job that this host had accepted
(runtasks) but not yet registered was dropped with "Failed to find worker".
handle_runtasks() now opens the same window before queuing the task and
closes it if queuing fails; the signal is delivered when the task runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
worker_signal_callback() sent SIGTERM for a KILL request without force and
SIGSTOP for any TSTP request. job_signal_to_deliver() (new job_signal.c) now
delivers the requested signal; only a forced terminate escalates to SIGKILL.
The apiserver's default signal for a plain stop becomes SIGSTOP (it was
TSTP and relied on the worker's substitution), so a plain stop still
suspends jobs that ignore SIGTSTP.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
preCheckJobRequest() returned from inside its job-name branch, so every
named or array job skipped the description, project and resreq checks,
resreq validation and the shell-limit conversion that follow. The name now
only sets the job count and array flag, and every submission runs the rest.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
deployment.md: auth.yaml now holds bcrypt hashes of the (still public)
dev credentials with startup warnings; JWT secret from XCE_JWT_SECRET or a
random per-process secret; logout revocation is per process;
security.trusted_proxies for reverse proxies.
security.md steering: JWT parsing, revocation, secret resolution,
trusted proxies, middleware abort rule, bcrypt and role-change rule.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: AddRolesForUser and DeleteRolesForUser let an rbac:control grant
narrower than all (e.g. scope self, an accepted domain type for rbac
permissions) change roles whenever the target user was the caller, so a
user holding only rbac:control/self could grant itself cluster_admin
(confirmed by TestAddRolesForUser_SelfScopeCannotGrantItselfRoles: HTTP
200 and cluster_admin assigned).
Fix: canChangeUserRoles requires an rbac:control grant with scope all
(checked over every grant, before deduplication prefers a narrower scope)
to grant or revoke roles for any user, the caller included; anything else
gets 403. Chosen over a "cannot grant permissions you do not hold" rule
because a scoped role change has no safe meaning here and the subset
check across domains would be complex and easy to get wrong.
Tests: TestAddRolesForUser_SelfScopeCannotGrantItselfRoles,
TestDeleteRolesForUser_SelfScopeRejected, TestUserRoleChanges_AllScopeAllowed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: Logout only deleted tokens/<user>/token, a file that token
validation (ValidateHandler / GetUsernameFromToken) never reads, so a
logged-out token stayed valid until its expiry (24 h by default).
Fix: an in-memory revocation set (controllers/token_revocation.go) keyed
by the token's jti (generateJWT now sets a random uuid jti, so two logons
in the same second give different tokens) or, for tokens without a jti,
by SHA-256 of the token; each entry is kept until the token's expiry and
expired entries are purged on every revocation. Logout revokes the token;
the shared ParseTokenClaims (used by ValidateHandler, RequirePermission
and handlers via GetUsernameFromToken) rejects revoked tokens. A missing
token file no longer fails logout. Scope: per process, not persisted and
not shared between apiservers.
Tests: TestLogout_RevokesToken (middleware), TestTokenRevocations_*,
TestIntegration_PostLogoutTokenRejected (replaces the design-gap test).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: config/app.yaml shipped secretKey "xce-api-key" and
resolveJWTSecret used it whenever XCE_JWT_SECRET was unset, so anyone
could forge tokens for any user on such a deployment.
Fix: resolveJWTSecret uses XCE_JWT_SECRET, else the config value, but
treats an empty value or the old public default as unset; then it
generates a random 256-bit per-process secret and InitConf logs a warning
(secret never logged): tokens do not survive a restart and apiservers
sharing one endpoint must set the same XCE_JWT_SECRET. The shipped
app.yaml no longer carries a secret. The controllers test secret is no
longer the old default.
Tests: TestResolveJWTSecret_NeverUsesKnownDefaultOrEmpty,
TestShippedAppYAML_NoJWTSecret, TestInitConf_NoJWTSecretGeneratesAndWarns,
TestInitConf_JWTSecretFromEnvNoWarning.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the router used gin.Default() without SetTrustedProxies, and gin
then trusts forwarding headers from every peer, so ClientIP() returned
whatever X-Forwarded-For a client sent: the IP whitelist and the IP bound
into JWTs could be spoofed.
Fix: new app.yaml option security.trusted_proxies (IP or CIDR list,
default empty = trust no proxy, validated at startup) passed to
SetTrustedProxies in Router(); an invalid list falls back to trusting
none. The nginx samples note that the proxy must be listed there.
Tests: TestRouter_IgnoresForwardedForByDefault,
TestRouter_HonoursForwardedForFromTrustedProxy, TestResolveTrustedProxies.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: ReturnError only writes the response and never aborts the gin
chain; RequirePermission (token error) and WhitelistIPMiddleware (invalid
client IP) returned after ReturnError without ctx.Abort(), so the route
handler still ran after the rejection.
Fix: ctx.Abort() after ReturnError at both sites (and, for clarity, on the
ValidateHandler body-check path, which validateRequestBody already
aborted). RequirePermission now answers a bad token with 401 instead of
500. ReturnError stays non-aborting (documented): handlers such as
ControlHost call it from helpers and then test ctx.IsAborted(), so a
global abort would change handler flows, and it adds nothing in handlers.
Tests: TestRequirePermission_AbortsOnInvalidToken,
TestWhitelistIPMiddleware_AbortsOnFailure.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the ValidateHandler parse callback (middleware isValidToken)
returned the secret without looking at token.Method, and
GetUsernameFromToken accepted any HMAC variant, so tokens signed with
HS384/HS512 were accepted although the server only signs HS256.
Fix: one JWTSigningMethod (HS256) used for signing, plus a shared
JWTKeyFunc (rejects any other alg) and JWTValidMethods parser option used
by both parse paths.
Tests: TestIsValidToken_PinsSigningAlgorithm (middleware),
TestGetUsernameFromToken_PinsSigningAlgorithm (controllers).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: ConfigFileProvider.Authenticate compared the stored plaintext
password with != (timing side channel, passwords readable from the file),
and config/auth.yaml shipped admin/admin, user1/user1 etc. as plaintext
with nothing reporting that a default credential was active.
Fix: auth.yaml passwords may be bcrypt hashes (golang.org/x/crypto/bcrypt,
v0.23.0 as already pinned in go.mod, now vendored); plaintext entries are
still accepted but compared in constant time (SHA-256 + subtle), and an
unknown user costs the same bcrypt work. The provider reports startup
warnings (SecurityWarner, logged by main) for every plaintext entry and for
every shipped default credential still active. The shipped auth.yaml and
auth.yaml.example now store bcrypt hashes of the same dev passwords, so
tests/regression/run.sh (admin/admin) keeps working. tools/encrypt gains
-action=hash to generate entries.
Tests: TestConfigFileProvider_BcryptPassword,
TestConfigFileProvider_SecurityWarnings, TestShippedAuthConfig_HashedDefaults.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: NewConfigFileProvider read cfg.ConfigFile.Users without checking
that the optional config_file block was present, so provider: config_file
with no block panicked with a nil-pointer dereference at startup.
Fix: return a clear configuration error when the block is missing.
Test: TestNewConfigFileProvider_Errors/missing_config_file_block.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: job_ctrl (job.c) sent kill/stop/resume only to allocs[0]->host.
The tasks on the other hosts are started through runtasks (by the head
worker for a parallel job, or by xrun as the MPI launcher's rsh agent)
and run under their own worker entry on each host; the head worker
signals only its local process group and forwards nothing, so they kept
running.
Fix: job_ctrl sends the request to every allocated host through
job_ctrl_to_host. The head decides the result as before; an unconnected
remote host is logged and skipped (its tasks belong to host failure
handling). Sends are non-blocking enqueues. Test:
testsuite/test_job_ctrl_hosts.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: job_ctrl (job.c) sent kill/stop/resume of a running job only when
the host's status was exactly 0, so any status bit (CLOSED,
CLOSED_BY_ADMIN, BUSY, LOCKED...) refused the request, and the refusal
returned -errno, i.e. whatever errno happened to hold (often 0 or
-ENOENT).
Fix: new job_ctrl_to_host sends the request whenever the worker is
connected (endpoint present, no XCE_WORKER_UNAVAIL); an unknown or
unconnected worker fails with -E_XCE_BAD_WORKER and an ERROR naming the
host. Nothing else changes for jobs on dead hosts. Test:
testsuite/test_job_ctrl_hosts.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: ControlJob (backend/clib.go) looked its content argument up as a
signal name, but the job control handlers pass a JSON JobControlEvent, so
the lookup always missed and every kill sent TERM (stop TSTP, resume
CONT); sig.force and ckpt.kill were hard-coded to 0 and the checkpoint
method was never set.
Fix: new pure jobCtrlParams (backend/convert_pure.go) decodes the event
(or a bare signal name/number), validates the signal against
JobSignalMapTable and returns an error instead of falling back, and maps
force_kill to xce_signal_t.force (kill) or xce_ckpt_options_t.kill
(checkpoint) plus the checkpoint method. ControlJob fills the C structs
from it. Tests: backend/jobctrl_params_test.go.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the apiserver writes file transfers to exdata as "transfer_files"
(controllers/jobs.go processJobRequest), but parse_exdata() only matched
"transfer_file", so the transfers were logged as an unknown key and
dropped.
Fix: parse_exdata() accepts "transfer_files" and still the older
"transfer_file"; if both are present the first list wins instead of
leaking it. New worker unit test test_exdata_parse.c parses the exdata
JSON the apiserver produces (transfers, env, interactive, limits).
Stage-in/out itself is still not implemented in the worker.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: modelJobToSubmit() filled only cmdline, execuser, launch_type and
pre/post-exec of job_startinfo_t, so the extended data the submit handler
builds (env, *_limit incl. -W run limit, file transfers, interactive mode),
the cwd and the exec home never reached the controller or the worker.
Fix: map the start info in a pure submitStartInfo() (exdata from
ExtendData, execcwd from ExecCWD or CWD, exechome from ExecHome) that
modelJobToSubmit copies into the C struct; decode exdata back in
cJobToModel/cJobArrayToModels so the store keeps ExtendData and CWD.
Reject extend_data the worker's parse_exdata() cannot take (not a JSON
object, over EXDATA_LENGTH-1 bytes) and drop an empty transfer list
instead of sending a null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both job-fields update paths, the core JOB_UPDATE event (core_events.c,
used by the live and sim drivers) and sched_update_existing_job()
(schedobj.c), set job->priority to the new queue's priority when a job
changed queue, while ingest gives every job 0. builtin_fcfs orders a
queue's jobs by job priority first, so a moved job overtook every job
already pending in the target queue (and snapshot restore, which sets 0,
disagreed with live for moved jobs).
Moving now only re-points job->queue; queue priority still orders the
queues themselves. test_job_move_priority.c moves a job into a priority-10
queue and checks it keeps priority 0 and does not take the only slot from
the queue's first job; test_job_update.c's in-place update case asserted
the old behaviour and now expects the job's priority to be kept.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only live's sched_init_config() (scheduler.c) registered the config's
named user and host groups into core.ugmgr/hgmgr. The simulator takes its
config from the replay snapshot (snapshot_load.c) and later pushes through
the core CONFIG_PUSH event (core_events.c); neither loaded groups, so a
limit whose users name a group ("ug1") never matched in sim, while the
same limit matched in live.
New xce_core_load_groups() (core.c) is the one loader, called by
sched_init_config(), the CONFIG_PUSH event and snapshot restore, each
before limit consumers are built. snapshot.c's field notes now say the
groups are rebuilt from the snapshot's config.
test_sim_group_limit_parity.c checks a ug1-scoped limit matches the same
job on both drivers and that CONFIG_PUSH alone registers ug1 and hg1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No scheduler code read XCE_WORKER_CLOSED / XCE_WORKER_CLOSED_BY_ADMIN:
sched_host_iter_next() (host_iter.c), the candidate-host source for both
the seqjob and the parjob allocator, filtered only on the select
expression and resource availability, so jobs kept landing on a host an
admin had closed (parjob's skip_bad_worker_ hook is empty and unused).
sched_host_iter_next() now skips a host with either closed bit before
opening its trace scope, which covers every new placement for serial and
parallel jobs on the live and simulation drivers. Accounting of jobs
already running on the host is unchanged. No new pend-reason code: a
closed host is simply not a candidate. test_host_admin_close.c covers a
serial and a parallel job on both fixture shapes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_queue_is_ready() in core.c was a `return 1` stub, so the picker
(builtin_fcfs and the legacy fcfs) took jobs from every queue and admin
queue inactivation never reached dispatch.
It now returns false for a queue with QUEUE_STAT_INACT_ADMIN or
QUEUE_STAT_INACT_WINDOW set, so an inactive queue's jobs stay pending.
QUEUE_STAT_CLOSED is deliberately not a gate (LSF semantics: closing only
blocks new submissions; already-pending jobs still dispatch). The status
reaches the core through the same queue add/update paths in both drivers,
so live and sim behave identically. test_queue_admin_control.c covers
open, closed and inactive queues on both fixture shapes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
27 cross-spec conflicts among the 30 pending specs (5 high), code gaps
against the five product goals and the no-shared-storage use case (silent
failures such as unenforced queue/host close and REST submit dropping job
limits, cwd, env and file transfers; inter-component and apiserver
security; file staging; monitoring and accounting; scheduling policy;
operations), and a tiered priority list (spec hygiene, P0 defects, P1
foundation specs, P2 features, P3 initiatives).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce-node-manager: a "Cross-Spec Interactions" section (pending a
requirements revision; approved criteria unchanged) with six points: a
supervision/probe module shared with service-supervision, node-manager
components vs service jobs, a lease-based provisioner role, role
declaration and worker stop on decommission for provisioned hosts, a
host-loss grace period longer than worker restart plus rejoin, and a future
service proxy under the reverse-proxy role. design.md Open Questions points
to it.
The service-supervision, host-failure-job-recovery, elastic-hosts,
resource-provisioner and service-endpoint briefs cross-reference the
matching points; the roadmap lists the xce-node-manager revision.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New briefs: provisioning-demand (the scheduler knows a provisionable
catalog, tries host templates as virtual hosts and license headroom after
real resources fail, finalizes demand and idle supply at session end;
simulator support is a hard requirement), resource-provisioner (new
daemon, several instances owning providers via controller leases,
self-describing providers, automatic policy, durable request/instance
state; src/autoscaler is prior art only), elastic-hosts (host templates,
expected hosts, drain and decommission), elastic-shared-resources
(borrowed license tokens change the shared total).
Roadmap: the new specs, plus updates to resource-distribution-mapping
(shared pools) and peer-authentication-hardening (provisioner role).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New briefs: host-failure-job-recovery (jobs on a lost host are requeued or
end EXIT after a grace period; stale copies fenced on rejoin; manual
requeue), service-supervision (profile kind service: restart policy and
liveness/readiness probes run by each replica's own worker, in-place
restart), service-replica-set (controller-held service object owning
independent replica jobs; event-driven replacement, scaling and rolling
update; no head agent, no promotion), service-endpoint (per-replica ports,
endpoints of ready replicas), job-resize (backlog, elastic jobs).
application-profile brief gains the service section; roadmap lists the new
specs, the existing-spec updates and the backlog item.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1934: every job-permission check read a single scope and compared it
only with "self", so a job:view or job:control grant scoped to self_ug acted
like "all" (list, get by ID, control), and self + self_ug collapsed to self.
userJobScope() now collects all of a user's grants of a job permission
before deduplication and adds them up (all; self; self_ug = the user and
everyone sharing a user group). The job list appends a submit_user filter
for narrower scopes, get-by-ID keeps covered rows in a new slice (removing
from the ranged slice skipped rows) and answers 403, both control paths
and job output check the owner against the scope. User groups now reach
the apiserver: the config push converts cluster_config_t.ugroups and
OnUserGroupsUpdated() replaces the new store (nested groups, all,
exclusions). The default basic_user now also sees its user-group mates'
jobs, as its configuration says.
bug-1935: project_name and license_project are accepted job filters, so
xcecli jobs list -P works.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
job-config-merge: spec initialized from its discovery brief.
project-config: requirements generated (9 requirements): nested projects
named by path, membership-based submit check with no inheritance and
overlapping membership, group-default then cluster-default project, open
mode until projects are defined, pending jobs held with a reason after
project changes, roll-up quotas, project visibility, a project scope for
job permissions covering the subtree and a project:view permission,
compatibility.
Roadmap: project-delegated-membership backlog item (runtime membership
edits by project managers, deferred).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The limit plugin called build_limit() with the host-group and
project-group managers swapped, so a limit's hosts: members were looked up
among the project groups (a host-group name became a literal host name and
the limit never applied) and its projects: members among the host groups.
Pass them in the order build_limit() declares.
New test test_limit_scopes_resolve_against_own_group_kind; the test's
main() now initializes the interned-string table that group names need.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
