The scaler is multi-threaded (monitor and demand threads); invoke() forked
and, in the child, logged, snprintf'd and setenv()'d before execl(), which
can deadlock on a lock another thread held at fork time and block the
caller in waitpid() forever.
run_script() now posix_spawn()s `sh <script> -f <in.json>`: stdout and
stderr go to the output file through file actions, the provider variables
(PRO_CONF_DIR, PRO_DATA_DIR, PRO_LOG_DIR, PROVIDER_NAME) are passed as the
child's envp (provider_env()), the child starts with an empty signal mask
and default dispositions, and waitpid() retries on EINTR.
The scaler unit suite (scaler/testsuite) was never built; it is now the
meson test "autoscaler test", linked with -Wl,--wrap=fork so it counts
fork() calls during a provider call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
RestartJobByID rebuilt the new job from the old job's row:
- it accepted asked_hosts and copied asked hosts, I/O paths and out_dir,
which no job gets (refused at submit since bug-1936); restart now
refuses asked_hosts with 400 and no longer carries those over;
- it read the JSONB columns chkpnt_info/extend_data as strings, but rows
keep models.Jsonb, so env/limits were lost and a checkpointed job never
restarted from its checkpoint; new rowJSON() reads both;
- it read "res_req" (the column is "resreq"), losing the resource
requirement, and never copied exec_home.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Design constraint (tech.md "Process Creation"): the controller, scheduler,
xcecd and anything that starts libxce threads never call fork, vfork,
forkpty, daemon, system, popen or uv_spawn; children start with posix_spawn()
or a systemd unit. A process proven single-threaded may fork, marked
/* xce-fork-ok: <why> */. Tests, harness, autoscaler (not finalized; bug-1969)
and the Go frontend are out of scope.
- scripts/check-no-fork.py scans src/{controller,scheduler,lib,worker}
(comment/string/#if 0 aware); runs as meson test "no-fork rule" in the
lib/controller/scheduler/worker suites and in the pre-commit hook.
- worker: drop xcecd's unused fork-based worker start paths
(process_manager_start_process, process_manager_start_worker; xcecd
starts xcecd_worker over D-Bus) and a dead #if 0 daemonize fork.
- xcecd_worker's fork/uv_spawn/system calls are marked: its call graph
from main(), callbacks included, reaches no pthread_create or libuv
thread-pool call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: CreateJob accepted in_file, out_file, err_file, out_dir,
asked_hosts, begin_time, term_time and transfer_files and returned
success, but neither the controller's submit data/start info nor the
worker carries any of them (the worker never stages transferred files),
so they were silently dropped. Exec home had no submit field at all, and
a relative cwd would have been resolved against the worker unit's
directory.
Fix: new preCheckSubmitOptions() runs right after binding, before file
uploads and ID allocation, and answers 400 naming every unsupported
option that is set; cwd and the new exec_home field must be absolute and
fit the worker's job_assignment_t buffers (1023 bytes). exec_home goes
to JobInfo.ExecHome and from there (submitStartInfo) to the start info's
exechome. num_files already reaches exdata as nofile_limit; a handler
test now pins it. The array %J/%I submit-time substitution test (9.4)
checks cwd instead of out_file/err_file, which are now refused. Swagger
regenerated with `swag init -g cmd/main.go -o docs`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the start info's execcwd/exechome reached xcecd and its worker-list
entry, but the MSG_JOB_ASSIGNED reply to the per-job xcecd_worker
(job_assignment_t) carried neither, and uv_proc_exec() spawned the job
with cwd NULL, so every job ran in the xcecd_worker unit's directory
whatever --cwd said, and exec home was never applied. parse_limits()
ignored the apiserver's "nofile_limit", so num_files never became an
rlimit.
Fix: job_assignment_t gets cwd/home, filled by the new
ipc_fill_job_assignment() (uv_ipc.c, which also NUL-ends every copied
string). xcecd_worker's job callback calls the new
uv_proc_enter_job_dirs() (uv_proc.c) before pre-exec and the job: it
exports HOME and chdir()s into cwd, falling back to /tmp with a warning
when the directory cannot be entered (the xcecd_worker runs as the job
user, so this is the user's own access). parse_limits() reads
nofile_limit into res_limits_t.nofile_limit and start_systemd_job() sets
LimitNOFILE like LimitNPROC. xcecd_virtual spawns no job and is
unaffected.
Tests: testsuite/test_job_dirs.c (assignment carries cwd/home, enter
cwd/home, fallback, a spawned job's pwd is the cwd) and
test_apiserver_nofile_limit_parsed in test_exdata_parse.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller sends job control to every host of a multi-host job
(bug-1942) as soon as the job is dispatched, but a remote host's runtasks
request is relayed by the controller (task.c do_runtasks, stateless) only
after the head worker has started and asked for it. A kill/stop that
reached a remote host first found no worker and no pending start, was
dropped ("Failed to find worker for job ID"), and the task started later
and kept running.
pending_signal now also holds "early" entries: handle_jobcontrol() records
a signal for a job with no task here (pending_signal_record_early), and
handle_runtasks() adopts it (pending_signal_start_task), so it is delivered
at RUNNING through the bug-1908 path and the task's exit is reported as
usual. Early entries are bounded (256) and expire after 60 s, pruned on
every early record and task start. A late control for a task that already
finished here (finished-job retention) is still dropped, and a head-host
start does not adopt an early entry, so neither can hit a later run of the
same jobid.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
limit_seed_config_impl() key_add()ed each pushed limit into the existing
table and returned at the first failure, so on any second push (live
config upgrade, or a simulator CONFIG_PUSH since bug-1964) the first
already-present name failed with -EEXIST: changed limits kept the old
definition and later new limits were never added.
A push now builds the whole new limit set into a fresh table and swaps it
in (all-or-nothing: on any build/register failure the previous set and its
instances stay). Usage instances are carried across: an instance of a kept
limit whose scoping and resource are unchanged is rebound to the new
definition; an instance of a removed limit, or of one whose per-consumer
scoping, resource or percent changed, is dropped and re-created on the next
match. Usage itself lives in the resource manager under the instance's
virtual resource, so counts survive either way. Not part of the snapshot
(a snapshot carries the config and reseeds from it).
The seed moves to plugins/limit_config.c (limit.c is near the 800-line
cap) with the private data type in plugins/limit_internal.h. Two REPL
limit-table fixtures relied on the additive behaviour (two single-limit
pushes) and now configure both limits in one push.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: parjob's allocation now draws candidate hosts from the shared
sched_host_iter_next() (host_iter.c) instead of its own
sched_collect_workers() loop, so skip_bad_worker_ lost its only caller
(the compiler warns it is unused). seqjob's copy was already removed when
it moved onto the shared iterators (b18c5f78).
Fix: remove it. Its only check (a NULL worker) is done by
sched_host_iter_next(), which is also where host filtering now lives
(closed hosts, bug-1939; unavailable and reconciling hosts, bug-1963).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: live applies every pushed config through sched_init_config(),
whose plugin_config_update() runs each plugin's config_hook (the limit
plugin builds its limits there). The simulator seeded the limit plugin
only at bootstrap (sim_bootstrap_limit_plugin()); the core's CONFIG_PUSH
event loaded groups, resources and queues but never told the plugins,
so a limit added during a replay never existed in sim.
Fix: one shared fan-out, plugin_config_update_core(core, config), which
live's plugin_config_update() now delegates to. The core's CONFIG_PUSH
calls a driver-registered xce_core_config_cb_t after loading groups (live
order), keeping the core free of plugin.c; the simulator registers the
fan-out in sim_bootstrap_limit_plugin(). limit_update_config() reads the
honestly-typed plugin->core instead of casting plugin->sched, so the hook
is safe on the sim core; the hand-built plugin fixtures in
test_limit_plugin.c/test_repl_limit_table.c now set plugin->core as
plugin_register() does. test_sim_limit_config_push_parity.c covers it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: sched_host_iter_next(), the one candidate-host source of the
seqjob and parjob allocators on both drivers, skipped only admin-closed
hosts (bug-1939). A host marked XCE_WORKER_UNAVAIL or
XCE_WORKER_RECONCILING was still offered, and its frozen resInUse
(bug-1799/1801) does not reject it, so new jobs were placed on it.
Fix: skip IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING hosts next to the
closed-host check. test_host_unavail_reconciling.c covers serial and
parallel jobs on the normal-mode and sim-mode fixtures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since the controller rejects submissions to closed or unknown queues
(E_XCE_QUEUE_CLOSED, E_XCE_BAD_QUEUE), the apiserver answered them as a 500
"xcecli_jobsubmit failed: -10016". SubmitJob now maps them to
backend.ErrQueueClosed / ErrQueueInvalid (numbers checked against errnox.h
at compile time), and the submit and restart handlers answer 400 naming
the queue, as the apiserver's own pre-check does. This covers a queue
closed between that check and the controller's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: handle_jobsubmit() never looked at the target queue, so a
submission that bypassed the apiserver's check (libxce, harness
xce_test, or a REST submit without a queue name) was accepted into a
closed queue, an unknown queue, or the reserved lost_and_found queue.
Fix: jobsubmit_check_queue() reads the queue's in-memory state under the
qmgr/queue read locks. handle_jobsubmit() calls it after the field
checks and before the in-flight jobid registry and any WAL append, for
single jobs and job arrays. It replies -E_XCE_QUEUE_CLOSED (new code,
"queue is closed") for a closed queue and -E_XCE_BAD_QUEUE for an
unknown or reserved one. Apply/replay (restore_job) is unchanged, so
committed submit facts for a now-closed queue still apply.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the apiserver and all components ran go test on a fixed list
(./store/... ./utils/... ./controllers/... ./backend/...), so the
middleware, router, cmd and models tests (several security tests among
them) never ran in /xce-build; a deliberately broken middleware test left
the build green.
Fix: run ./... (vendor/ is excluded by the pattern) in both places.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the handler annotations referenced types that do not exist
(models.Error, ErrorResponse, JobsResponse, JobSignalRequest,
JobControlResponseSummary, unqualified JobSubmitRequest), had malformed
@Param/@Failure lines and /api/v1 paths, and there was no general API
info, so `swag init -g cmd/main.go` failed and docs/ stayed a stale stub.
Fix: use the real response envelopes (JsonStruct / JsonErrStruct) and
request models, the real routes under @BasePath /xce/v1, add general info
to cmd/main.go, and regenerate docs/ with `swag init -g cmd/main.go -o
docs`; the spec now has /jobs/{jobids}/control and /jobs/control.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: preCheckJobRequest (controllers/jobs.go) ran the reserved-name,
existence and closed checks only when a queue name was given; an empty
name was set to "normal" and skipped them, so jobs were accepted into a
closed default queue.
Fix: default the name first, then run the same checks for every
submission. Two existing tests now seed the default queue. (The controller
side of bug-1960 is fixed separately.)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: isExcludedPath (middleware/validatehandler.go) dropped the last
character of every excluded entry and prefix-matched, so /ping excused any
path starting with /pin (e.g. /pinX) from JWT validation, while
/swagger/*any (trimmed to /swagger/*an) never matched.
Fix: entries match exactly; an entry whose last segment is a gin catch-all
(/swagger/*any) matches any path under the part before the "*".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: validateHost (controllers/hosts.go) wrote a 403 via ReturnError
without aborting, so ControlHost's ctx.IsAborted() check never fired and it
went on to write a second (202) response; the permission was also checked
once per host and not at all for an empty host list.
Fix: validateHost writes no response; it checks host:control once up front
and returns an error, on which ControlHost writes a single 403 and returns.
For "all", the host list is resolved before the one validateHost call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: RoleController.CreateRole, AddPermissionsForRole and
DeleteRoleAndPermissions relied on RequirePermission("rbac:control"), which
accepts any scope, so a user with rbac:control scoped to self could add
cluster:control to a role it holds.
Fix: checkRoleChangePermission() in controllers/roles.go refuses with 403
unless one of the caller's grants (checked before deduplication) is
rbac:control with scope all, reusing canChangeUserRoles from bug-1952.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control API already takes `signal` (kill only: that signal once,
instead of the graceful INT, TERM, KILL sequence) and `force_kill`
(SIGKILL at once); xcecli could not send either. `jobs kill` now has
-s/--signal <name|number> (names in any case, SIG prefix optional, sent
upper case without it) and --force, in both the job-ID and the filter
request; using both at once, or an empty signal, is refused before any
request. stop and resume get neither (the server rejects a signal there).
Kill help and docs/xce.md "Killing and stopping jobs" document the flags.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli logout only removed the token from the local config, so the token
stayed valid on the server until it expired. It now first sends
POST /xce/v1/auth/logout with the bearer token (APIClient.Logout), which
revokes it server side since bug-1946. Best effort: if the server is
unreachable or rejects the token, the local login info is still cleared
and a warning says the revocation could not be confirmed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- src/docs/xce.md: explicit signal and force kill are REST-only (signal,
force_kill); xcecli has no signal option yet.
- src/docs/deployment.md: app.yaml row lists the job block
(max_array_size, terminate_interval) and trusted proxies.
- xcecli jobs kill/stop/resume help describes the signal sequences.
- REST: control endpoint annotations and the signal/force_kill model fields
describe the behaviour (swagger output not regenerated: swag fails on an
unrelated annotation).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The loader now stores the effective interval (10 s when unset or 0, LSF's
JOB_TERMINATE_INTERVAL default) and refuses a negative one; app_schema.json
describes the setting; src/docs/xce.md documents kill/stop behaviour and
the interval.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A plain kill sent one SIGTERM and never escalated, and a plain stop sent
one uncatchable SIGSTOP. Following LSF's TERMINATE and SUSPEND actions, a
plain kill now sends SIGINT, then SIGTERM, then SIGKILL, and a plain stop
sends SIGTSTP (the job can catch it, prepare and stop itself), then SIGSTOP
if it has not stopped; each next signal follows after the grace period and
only while the job is alive (or not yet stopped). A resume cancels a
pending stop; the job's exit ends any sequence. An explicit signal is
delivered once; a force kill is SIGKILL at once.
- JobSignal gains grace (capnp @2, xce_signal_t.grace): > 0 asks for the
sequence, in seconds between signals.
- apiserver: job.terminate_interval (default 10 s, LSF's
JOB_TERMINATE_INTERVAL); jobCtrlParams sends kill = SIGKILL and stop =
SIGSTOP with that grace when no signal is given; the handler no longer
pre-fills a default signal.
- worker: grace travels through the job-control task, the kept-signal table
(pending_signal_*_ex) and the IPC message; the job process runs the
sequence (job_signal_seq_*, uv timer) and checks stop state in /proc.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only a head-host job start opened the pending-signal window (bug-1908), so a
kill for a remote task of a multi-host job that this host had accepted
(runtasks) but not yet registered was dropped with "Failed to find worker".
handle_runtasks() now opens the same window before queuing the task and
closes it if queuing fails; the signal is delivered when the task runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
worker_signal_callback() sent SIGTERM for a KILL request without force and
SIGSTOP for any TSTP request. job_signal_to_deliver() (new job_signal.c) now
delivers the requested signal; only a forced terminate escalates to SIGKILL.
The apiserver's default signal for a plain stop becomes SIGSTOP (it was
TSTP and relied on the worker's substitution), so a plain stop still
suspends jobs that ignore SIGTSTP.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
preCheckJobRequest() returned from inside its job-name branch, so every
named or array job skipped the description, project and resreq checks,
resreq validation and the shell-limit conversion that follow. The name now
only sets the job count and array flag, and every submission runs the rest.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
deployment.md: auth.yaml now holds bcrypt hashes of the (still public)
dev credentials with startup warnings; JWT secret from XCE_JWT_SECRET or a
random per-process secret; logout revocation is per process;
security.trusted_proxies for reverse proxies.
security.md steering: JWT parsing, revocation, secret resolution,
trusted proxies, middleware abort rule, bcrypt and role-change rule.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: AddRolesForUser and DeleteRolesForUser let an rbac:control grant
narrower than all (e.g. scope self, an accepted domain type for rbac
permissions) change roles whenever the target user was the caller, so a
user holding only rbac:control/self could grant itself cluster_admin
(confirmed by TestAddRolesForUser_SelfScopeCannotGrantItselfRoles: HTTP
200 and cluster_admin assigned).
Fix: canChangeUserRoles requires an rbac:control grant with scope all
(checked over every grant, before deduplication prefers a narrower scope)
to grant or revoke roles for any user, the caller included; anything else
gets 403. Chosen over a "cannot grant permissions you do not hold" rule
because a scoped role change has no safe meaning here and the subset
check across domains would be complex and easy to get wrong.
Tests: TestAddRolesForUser_SelfScopeCannotGrantItselfRoles,
TestDeleteRolesForUser_SelfScopeRejected, TestUserRoleChanges_AllScopeAllowed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: Logout only deleted tokens/<user>/token, a file that token
validation (ValidateHandler / GetUsernameFromToken) never reads, so a
logged-out token stayed valid until its expiry (24 h by default).
Fix: an in-memory revocation set (controllers/token_revocation.go) keyed
by the token's jti (generateJWT now sets a random uuid jti, so two logons
in the same second give different tokens) or, for tokens without a jti,
by SHA-256 of the token; each entry is kept until the token's expiry and
expired entries are purged on every revocation. Logout revokes the token;
the shared ParseTokenClaims (used by ValidateHandler, RequirePermission
and handlers via GetUsernameFromToken) rejects revoked tokens. A missing
token file no longer fails logout. Scope: per process, not persisted and
not shared between apiservers.
Tests: TestLogout_RevokesToken (middleware), TestTokenRevocations_*,
TestIntegration_PostLogoutTokenRejected (replaces the design-gap test).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: config/app.yaml shipped secretKey "xce-api-key" and
resolveJWTSecret used it whenever XCE_JWT_SECRET was unset, so anyone
could forge tokens for any user on such a deployment.
Fix: resolveJWTSecret uses XCE_JWT_SECRET, else the config value, but
treats an empty value or the old public default as unset; then it
generates a random 256-bit per-process secret and InitConf logs a warning
(secret never logged): tokens do not survive a restart and apiservers
sharing one endpoint must set the same XCE_JWT_SECRET. The shipped
app.yaml no longer carries a secret. The controllers test secret is no
longer the old default.
Tests: TestResolveJWTSecret_NeverUsesKnownDefaultOrEmpty,
TestShippedAppYAML_NoJWTSecret, TestInitConf_NoJWTSecretGeneratesAndWarns,
TestInitConf_JWTSecretFromEnvNoWarning.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the router used gin.Default() without SetTrustedProxies, and gin
then trusts forwarding headers from every peer, so ClientIP() returned
whatever X-Forwarded-For a client sent: the IP whitelist and the IP bound
into JWTs could be spoofed.
Fix: new app.yaml option security.trusted_proxies (IP or CIDR list,
default empty = trust no proxy, validated at startup) passed to
SetTrustedProxies in Router(); an invalid list falls back to trusting
none. The nginx samples note that the proxy must be listed there.
Tests: TestRouter_IgnoresForwardedForByDefault,
TestRouter_HonoursForwardedForFromTrustedProxy, TestResolveTrustedProxies.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: ReturnError only writes the response and never aborts the gin
chain; RequirePermission (token error) and WhitelistIPMiddleware (invalid
client IP) returned after ReturnError without ctx.Abort(), so the route
handler still ran after the rejection.
Fix: ctx.Abort() after ReturnError at both sites (and, for clarity, on the
ValidateHandler body-check path, which validateRequestBody already
aborted). RequirePermission now answers a bad token with 401 instead of
500. ReturnError stays non-aborting (documented): handlers such as
ControlHost call it from helpers and then test ctx.IsAborted(), so a
global abort would change handler flows, and it adds nothing in handlers.
Tests: TestRequirePermission_AbortsOnInvalidToken,
TestWhitelistIPMiddleware_AbortsOnFailure.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the ValidateHandler parse callback (middleware isValidToken)
returned the secret without looking at token.Method, and
GetUsernameFromToken accepted any HMAC variant, so tokens signed with
HS384/HS512 were accepted although the server only signs HS256.
Fix: one JWTSigningMethod (HS256) used for signing, plus a shared
JWTKeyFunc (rejects any other alg) and JWTValidMethods parser option used
by both parse paths.
Tests: TestIsValidToken_PinsSigningAlgorithm (middleware),
TestGetUsernameFromToken_PinsSigningAlgorithm (controllers).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: ConfigFileProvider.Authenticate compared the stored plaintext
password with != (timing side channel, passwords readable from the file),
and config/auth.yaml shipped admin/admin, user1/user1 etc. as plaintext
with nothing reporting that a default credential was active.
Fix: auth.yaml passwords may be bcrypt hashes (golang.org/x/crypto/bcrypt,
v0.23.0 as already pinned in go.mod, now vendored); plaintext entries are
still accepted but compared in constant time (SHA-256 + subtle), and an
unknown user costs the same bcrypt work. The provider reports startup
warnings (SecurityWarner, logged by main) for every plaintext entry and for
every shipped default credential still active. The shipped auth.yaml and
auth.yaml.example now store bcrypt hashes of the same dev passwords, so
tests/regression/run.sh (admin/admin) keeps working. tools/encrypt gains
-action=hash to generate entries.
Tests: TestConfigFileProvider_BcryptPassword,
TestConfigFileProvider_SecurityWarnings, TestShippedAuthConfig_HashedDefaults.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
