Cause: Logout only deleted tokens/<user>/token, a file that token
validation (ValidateHandler / GetUsernameFromToken) never reads, so a
logged-out token stayed valid until its expiry (24 h by default).
Fix: an in-memory revocation set (controllers/token_revocation.go) keyed
by the token's jti (generateJWT now sets a random uuid jti, so two logons
in the same second give different tokens) or, for tokens without a jti,
by SHA-256 of the token; each entry is kept until the token's expiry and
expired entries are purged on every revocation. Logout revokes the token;
the shared ParseTokenClaims (used by ValidateHandler, RequirePermission
and handlers via GetUsernameFromToken) rejects revoked tokens. A missing
token file no longer fails logout. Scope: per process, not persisted and
not shared between apiservers.
Tests: TestLogout_RevokesToken (middleware), TestTokenRevocations_*,
TestIntegration_PostLogoutTokenRejected (replaces the design-gap test).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: config/app.yaml shipped secretKey "xce-api-key" and
resolveJWTSecret used it whenever XCE_JWT_SECRET was unset, so anyone
could forge tokens for any user on such a deployment.
Fix: resolveJWTSecret uses XCE_JWT_SECRET, else the config value, but
treats an empty value or the old public default as unset; then it
generates a random 256-bit per-process secret and InitConf logs a warning
(secret never logged): tokens do not survive a restart and apiservers
sharing one endpoint must set the same XCE_JWT_SECRET. The shipped
app.yaml no longer carries a secret. The controllers test secret is no
longer the old default.
Tests: TestResolveJWTSecret_NeverUsesKnownDefaultOrEmpty,
TestShippedAppYAML_NoJWTSecret, TestInitConf_NoJWTSecretGeneratesAndWarns,
TestInitConf_JWTSecretFromEnvNoWarning.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the router used gin.Default() without SetTrustedProxies, and gin
then trusts forwarding headers from every peer, so ClientIP() returned
whatever X-Forwarded-For a client sent: the IP whitelist and the IP bound
into JWTs could be spoofed.
Fix: new app.yaml option security.trusted_proxies (IP or CIDR list,
default empty = trust no proxy, validated at startup) passed to
SetTrustedProxies in Router(); an invalid list falls back to trusting
none. The nginx samples note that the proxy must be listed there.
Tests: TestRouter_IgnoresForwardedForByDefault,
TestRouter_HonoursForwardedForFromTrustedProxy, TestResolveTrustedProxies.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: ReturnError only writes the response and never aborts the gin
chain; RequirePermission (token error) and WhitelistIPMiddleware (invalid
client IP) returned after ReturnError without ctx.Abort(), so the route
handler still ran after the rejection.
Fix: ctx.Abort() after ReturnError at both sites (and, for clarity, on the
ValidateHandler body-check path, which validateRequestBody already
aborted). RequirePermission now answers a bad token with 401 instead of
500. ReturnError stays non-aborting (documented): handlers such as
ControlHost call it from helpers and then test ctx.IsAborted(), so a
global abort would change handler flows, and it adds nothing in handlers.
Tests: TestRequirePermission_AbortsOnInvalidToken,
TestWhitelistIPMiddleware_AbortsOnFailure.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the ValidateHandler parse callback (middleware isValidToken)
returned the secret without looking at token.Method, and
GetUsernameFromToken accepted any HMAC variant, so tokens signed with
HS384/HS512 were accepted although the server only signs HS256.
Fix: one JWTSigningMethod (HS256) used for signing, plus a shared
JWTKeyFunc (rejects any other alg) and JWTValidMethods parser option used
by both parse paths.
Tests: TestIsValidToken_PinsSigningAlgorithm (middleware),
TestGetUsernameFromToken_PinsSigningAlgorithm (controllers).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: ConfigFileProvider.Authenticate compared the stored plaintext
password with != (timing side channel, passwords readable from the file),
and config/auth.yaml shipped admin/admin, user1/user1 etc. as plaintext
with nothing reporting that a default credential was active.
Fix: auth.yaml passwords may be bcrypt hashes (golang.org/x/crypto/bcrypt,
v0.23.0 as already pinned in go.mod, now vendored); plaintext entries are
still accepted but compared in constant time (SHA-256 + subtle), and an
unknown user costs the same bcrypt work. The provider reports startup
warnings (SecurityWarner, logged by main) for every plaintext entry and for
every shipped default credential still active. The shipped auth.yaml and
auth.yaml.example now store bcrypt hashes of the same dev passwords, so
tests/regression/run.sh (admin/admin) keeps working. tools/encrypt gains
-action=hash to generate entries.
Tests: TestConfigFileProvider_BcryptPassword,
TestConfigFileProvider_SecurityWarnings, TestShippedAuthConfig_HashedDefaults.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: NewConfigFileProvider read cfg.ConfigFile.Users without checking
that the optional config_file block was present, so provider: config_file
with no block panicked with a nil-pointer dereference at startup.
Fix: return a clear configuration error when the block is missing.
Test: TestNewConfigFileProvider_Errors/missing_config_file_block.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: ControlJob (backend/clib.go) looked its content argument up as a
signal name, but the job control handlers pass a JSON JobControlEvent, so
the lookup always missed and every kill sent TERM (stop TSTP, resume
CONT); sig.force and ckpt.kill were hard-coded to 0 and the checkpoint
method was never set.
Fix: new pure jobCtrlParams (backend/convert_pure.go) decodes the event
(or a bare signal name/number), validates the signal against
JobSignalMapTable and returns an error instead of falling back, and maps
force_kill to xce_signal_t.force (kill) or xce_ckpt_options_t.kill
(checkpoint) plus the checkpoint method. ControlJob fills the C structs
from it. Tests: backend/jobctrl_params_test.go.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: modelJobToSubmit() filled only cmdline, execuser, launch_type and
pre/post-exec of job_startinfo_t, so the extended data the submit handler
builds (env, *_limit incl. -W run limit, file transfers, interactive mode),
the cwd and the exec home never reached the controller or the worker.
Fix: map the start info in a pure submitStartInfo() (exdata from
ExtendData, execcwd from ExecCWD or CWD, exechome from ExecHome) that
modelJobToSubmit copies into the C struct; decode exdata back in
cJobToModel/cJobArrayToModels so the store keeps ExtendData and CWD.
Reject extend_data the worker's parse_exdata() cannot take (not a JSON
object, over EXDATA_LENGTH-1 bytes) and drop an empty transfer list
instead of sending a null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1934: every job-permission check read a single scope and compared it
only with "self", so a job:view or job:control grant scoped to self_ug acted
like "all" (list, get by ID, control), and self + self_ug collapsed to self.
userJobScope() now collects all of a user's grants of a job permission
before deduplication and adds them up (all; self; self_ug = the user and
everyone sharing a user group). The job list appends a submit_user filter
for narrower scopes, get-by-ID keeps covered rows in a new slice (removing
from the ranged slice skipped rows) and answers 403, both control paths
and job output check the owner against the scope. User groups now reach
the apiserver: the config push converts cluster_config_t.ugroups and
OnUserGroupsUpdated() replaces the new store (nested groups, all,
exclusions). The default basic_user now also sees its user-group mates'
jobs, as its configuration says.
bug-1935: project_name and license_project are accepted job filters, so
xcecli jobs list -P works.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller computed run usage (CPU time, memory, CPU peak and
efficiency, LSF bjobs -l definitions) but the capnp Job struct had no
field for it, so the job pushed to the apiserver never carried it, and
snapshots (the same struct) dropped it: a job restored from a checkpoint
lost its accumulated usage.
- controller.capnp: Job.runUsage @13 (JobRunusage); old entries decode
with no run usage.
- apiserver: map run_usage into JobInfo; max_mem, avg_mem, used_mem,
cpu_time, peak_cpu, peak_cpu_time, peak_cpu_duration, mem_efficiency,
avg_cpu_efficiency, peak_cpu_efficiency become queryable columns.
- xcecli: -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK,
CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now
renders too), and a MEMORY USAGE / CPU USAGE block in jobs list -l.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- apiserver peek (parseJobIDParam): <array-id>[k] now resolves to the
element; it used to drop [k] and look up the array identity itself
(and panicked in parseArrayName on some inputs). A job number still
names that job exactly (an element by its own XCE_JOBID); ranges and
invalid selections are rejected.
- apiserver restart: the decimal-only pre-check (isValidJobId, now
unused and removed) rejected the written form c-a-ts-seq[k];
ParseIdentitySelection, already used for the lookup, validates it.
- xcecli jobs restart converts its argument with ToAPIJobID like the
control commands, so written IDs work and invalid ones fail locally.
- docs: the peek/restart caveat in the Job arrays section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It logged in to a hard-coded http://localhost:8080 with fixed
credentials, so 'go test ./...' failed with connection refused. It now
skips unless XCE_LOAD_TEST_URL is set, with the credentials in
XCE_LOAD_TEST_USER/XCE_LOAD_TEST_PASSWORD; 'go test ./...' is green.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Formatting only (gofmt -w on the 29 non-vendored, non-generated files
gofmt -l reported); swag-generated apiserver/docs/docs.go left as is.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
RestartJobByID looked the job up with a job_id CompCondition, which the
store never evaluates, and restarted jqueryReply[0] -- the first stored
job -- for any ID, including unknown ones. It now parses the ID with
ParseIdentitySelection, fetches the rows by key and matches them with
foundJob, as ControlJobByID does: an ordinary job or <array-id>[k] is
restarted, an unknown job or element is not found, and a bare array
identity (several elements) is refused. Restart goes through the
generateJobID/submitJob seams so the handler fixture can test it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follows page_token with limit=3 over the handler fixture. Before
efff6b65 the token's float-rounded array-element ID matched no row and
every page restarted at row 0 (the original repeat-the-first-page
report); it now pages through every row exactly once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
findPageTokenIndex decoded the token's unique_job_id into a float64;
array element IDs set bit 62, where float64 spacing is 1024, so paging
skipped rows or restarted at row 0 -- an endless loop for callers that
page until the token is empty (control/query of arrays over 1000
elements). Decode with UseNumber and keep integers as int64.
Found by the job-array-one-based-index feature validation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1904: both scheduler JOB_UPDATE paths parsed "" as-is:
parse_resreq() returns success with a PRIMARY container whose body is
NULL, and apply_job_task_count() wrote through it. Use ingest's default
("type == any") instead.
bug-1906: xcecli jobs list sent local selection-parse errors through
ExtractErrorMessage, which strips brackets and the quoted selection;
report them as written.
Also logs bug-1905 (xcecd_virtual ignores kill of a running job). All
three found by job-array-one-based-index task 10.2's live REST flow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The comment still described arraySize > 1 as the array path; since task
9.1 idgen() rejects any size other than 1 and arrays use
GenerateArrayJobID.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Submitting an array prints the server's message with the array ID in
written form (no [k]) and the element count, e.g.
'Job <0-0-...> is submitted to queue <normal> (3 elements).'; ordinary
submits print exactly as before. The client model mirrors array_size.
Submit help documents name[1-N] and name[1], numbering from 1 and the
API-server limit (default 1000, at most 32767).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ID lists split only at commas outside brackets and every selection is
validated before any request (control allows identity ranges, listing
refuses them). A raw -J value with brackets is checked before the
wildcard mapping, and the search rule and * stripping apply to the
name part only (-J 'foo*[1]' lists element 1 of arrays whose name
starts with foo); values without brackets map exactly as before.
Control results print each item's own job ID when the server sends it,
so a pre-upgrade element 0 shows as <array-id>[0]. Control and list
help show <array-id>[k] and <array-id>[a-b]; list help now says -J.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli's format package gains ParseSelection (written or decimal key
with optional [k]/[a-b], the server's grammar and check order; 5.8 and
5.7 also on decimal keys with [...], so an upgraded client protects
itself against an older server; query ranges refused after the other
checks), the bracket-aware SplitSelections, the raw -J
ParseNameSelection and Selection.APIKey. ParseJobID is bounded;
ToAPIJobID(s, allowRange) no longer passes unrecognised text through;
peek keeps today's behaviour via ToAPIJobIDLegacy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Filter-based ControlJob validates a bracketed -J value before fetching
(400 with the parser's message; unbracketed names, including commands
with spaces, unchanged). FetchJobList evaluates bracketed job_name
entries through the selection predicates instead of unevaluated
compound conditions, so -J 'foo[1]' acts on element 1 of every array
named foo and never on ordinary jobs of that name. The most-recent-job
rule applies only without an element number and without all-jobs, and
treats the element of a one-element array like an ordinary job
(isSingleElementArray, a documented approximation). The old
index-section parser is deleted. Completes bug-1898 on the owned paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ControlJobByID parses every selection with ParseIdentitySelection before
any store access (first rejection: 400, nothing dispatched), collects
the jobs to act on de-duplicated by unique_job_id plus one not-found
result per missing number per selection (carrying the would-be element
ID), validates each job once after the loop and dispatches each
accepted job once. foundJob gains an explicit-index flag so
<array-id>[0] matches only element 0 and never falls back to the whole
array or an ordinary job. Fixes bug-1899 (earlier selections were
re-dispatched for every later one).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GetJobsInfo validates every filter entry in request order before any
store access (including the exec_hosts lookup) and GetJobByIDs every key
before querying; the first rejection returns 400 with the parser's
message. Bracketed job_id/job_name entries are evaluated as row
predicates before sorting and pagination, so job_id:eq:X[2] lists only
element 2, GET /jobs/X[1-3] lists elements 1-3 and <ordinary-id>[0]
lists nothing. The unevaluated compound conditions and the old index
parsing are gone from both handlers. Fixes bug-1898 on the query paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
BuildSelectionPredicates converts each job_id/job_name filter entry
containing a bracketed value into one row predicate (values OR'ed,
entries AND'ed with each other and the plain conditions), removing only
that entry. Operators are honoured instead of dropped (bug-1898):
eq/in/between select, notin excludes, name pattern operators reuse
today's condition semantics on the name part, and ne / comparison /
null operators keep today's rejection. Element selections match only
array elements in range; bare identities match the whole array and any
ordinary job; an empty name under exact operators matches nothing.
Bare written keys are normalised to decimal on every path. Not yet
wired into the handlers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ParseIdentitySelection (strict written form or base-10 decimal key,
optional [k] or [a-b]) and ParseNameSelection (name part may be empty,
no whitespace or brackets) apply the shared selection grammar and check
order (5.16 syntax, 5.14 width, 5.17 marked decimal without [k], 5.8
alignment only with [...], 5.7 element bound, 5.5 reversed), reporting
only the first failure with a rule tag and a message quoting the
selection. Runs the shared selection-cases table plus REST-only rows.
Not yet wired into the handlers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
preCheckJobRequest returns (n, isArray, err): a bracketed job name is
parsed by ParseSubmitIndexExpr first, so name[1-N] and name[1] create
arrays (one-element arrays included) allocated with
generateArrayJobID, stored under the bare name; expression errors are
answered 400 with their specific message and create nothing; names
without brackets take the unchanged createArrayName path. The response
names the array (jobid = array ID, new array_size, "(n elements)"),
submit-time %I/%J use the first element, and the submit and dispatch
logs identify arrays by array ID. Fixes bug-1896 for REST and xcecli.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ParseSubmitIndexExpr accepts only name[1-N] and name[1] (leading zeros
allowed) and otherwise reports the lowest-numbered broken rule with its
specific message: interactive arrays (2.1), invalid name or malformed
expression (2.2), lists/steps (2.3), %N (2.4), start 0 (2.5), reversed
(2.6), start above 1 (2.7), above the limit (2.8). Oversized numbers
saturate for the bound checks while reversal compares the digit
strings, so the rule order holds for any input. Not yet wired into
CreateJob (task 6.2).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CreateJob's ID generation and submission and both control dispatch
sites now go through package variables (generateJobID,
generateArrayJobID, submitJob, controlJob) that default to the real
backend/services functions, so production behaviour is unchanged. A new
non-cgo handler test file provides a reusable fixture: it saves and
restores every global it touches, sets up auth/RBAC, seeds jobs and
their queues, and records each dispatched job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
QueryOptions gains Predicates; store.ExecuteQuery keeps a row only if
every predicate returns true, after the plain conditions and before
sort, pagination and field selection, so page boundaries and tokens
count only matching rows. Nil/empty predicates leave every existing
query unchanged. Foundation for evaluating element selections
(bug-1898).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
job.max_array_size is resolved by a pure resolveMaxArraySize: unset or 0
gives 1000, 1..32767 is used as is, values above 32767 are clamped to
32767 with a startup warning naming the configured and effective value,
and a negative value refuses startup. The old >2147483646 rejection is
gone; the shipped config documents the bounds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GenerateArrayJobID wraps libxce idgen_array (first element, index 1;
one-element arrays included) with a non-cgo stub; pure helpers
ParseHumanJobIDStrict (written identity with exact field-width checks,
uint64 overflow reported as a width error), IsArrayCapableKey and
MakeElementID (bit-identical to libxce jobsel_element_id).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The apiserver's job store learned of a new job only from the
controller's asynchronous OnJobCreated / array push, so a 'jobs kill'
(or query) straight after 'jobs submit' could answer 'job not found'
for the job just accepted (2 of 4 immediate kills).
services.SubmitJob now records the accepted job right after the
controller's reply (which comes only after the job is committed and
applied): one record per element, base+i for arrays, stored with the new
DataStore.PutIfAbsent and marked JobInfo.Provisional (json:"-", not a
column). An OnJobCreated that already arrived is never overwritten; the
push that arrives later replaces the provisional record -- including in
dispatchJobArray, whose duplicate check no longer treats a provisional
element as already known.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli prints a submitted job's ID in the dashed
{cluster}-{apiserver}-{timestamp}-{sequence} form (format.FormatJobID,
since d6008ead), but run.sh still matched 'Job <[0-9]+>' -- so 'jobs
submit' always failed and 'jobs kill' was skipped (28/1/1). It also
extracted the ID with [0-9]{10,}, which would have picked out only the
timestamp. Match the dashed form and take the whole ID from between the
angle brackets; 'jobs kill' accepts it back.
'jobs submit' reads a job script from stdin whenever stdin is a pipe,
so the suite hung forever when run from a script or CI with an open
pipe; the submit now reads /dev/null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gitea #47: `xcecli jobs submit -R "rusage[mem=100000000a0]"` was accepted
with a normal "Job <...> is submitted" reply. The apiserver only
length-checked resreq; the real PEG grammar ran later, asynchronously, in
the scheduler's core_build_job(), which logged a server-side WARN and
dropped the job -- it then sat PEND forever with no sign of why.
- libxce: xce_check_rusage_syntax(resreq, &errmsg) runs parse_rusage() on
every rusage[...] clause (extracted the way section.peg does: up to the
first ']'), so primary, compound and alternative requests are all
covered. No resource manager needed, so select[]/order[] resource names
are still only checked by the scheduler (the documented scope).
- apiserver/backend: ValidateResreqSyntax() cgo wrapper (resreq_check.go);
the !cgo stub returns nil (no job can be submitted without CGO anyway).
- controllers/jobs.go preCheckJobRequest(): calls it after the length
check via a package var, so a malformed request is refused up front.
Tests: lib test_check_rusage_syntax (RED before: the reported input was
accepted) and backend TestValidateResreqSyntax (RED before) through the
real grammar; controllers TestPreCheckJobRequest_RejectsMalformedResreq
checks the call site with an injected validator (written with the call
site, so no separate RED). libxce, apiserver backend (cgo) and controllers
tests pass. apiserver/middleware TestLoadJobs fails here and on the
baseline alike: it needs a live apiserver on :8080.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ControlJob() set xce_jobctrl_t.action to the job_ctrl_type_t action code
(kill/stop/resume/checkpoint), but that field is actually the capnp
JobControl_u_which union discriminant -- a different enum entirely. This
made kill/resume/checkpoint hit request.c's -EINVAL default case, and made
stop collide with the checkpoint discriminant and read through the wrong
union member. Live-verified against a real deployed cluster: kill/stop/
resume/checkpoint now behave correctly with no apiserver crash.
Also additively accept a human-readable job ID ("{cluster}-{apiserver}-
{timestamp}-{seq}") on the job control endpoint, alongside the existing
numeric form xcecli already translates to -- lets a caller hit the REST
API directly without reimplementing xcecli's ID packing. Live-verified via
a direct curl call with the human-readable ID.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
modelJobToSubmit() never set job_startinfo_t.execuser when converting
a submitted job to the C struct sent to the controller/worker, so
every job ran as the worker's 'nobody' default and failed to exec
(exit 126). Default execuser to the submitting user via a new
resolveExecUser() pure helper, and free the newly allocated C string.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
NJOBS/RUN/SSUSP/USUSP (xcecli hosts list -l) and EXEC_HOST (xcecli jobs
list) always showed a hardcoded 1 slot per host regardless of a job's
actual per-host allocation -- rawJobToModel()'s ExecHosts JSONB builder
never read the real per-host "slots" resource from the C job allocation,
so the already-correct downstream slot-counting logic (computeHostSlotUsage,
bug-1743) and EXEC_HOST's "N*host" formatter never had real data to work
with. convert.go now extracts each host's real slots allocation via a new
CGO indexing helper, mirroring the scheduler's own get_nslots_in_alloc()
convention.
Also wires hosts list -l's "CURRENT LOAD USED FOR SCHEDULING" table (tmp/
swap/mem/slots and friends) to the host's real live resource values --
previously entirely hardcoded placeholder strings, never connected to any
backend data.
The terminal step: src/frontend/Makefile (194 lines, the second,
independently-maintained build/test path this whole spec exists to
eliminate) is deleted. Re-ran the full task 4.1 validation suite with
the file genuinely absent (build, meson test --suite apiserver, and
-- going beyond citing task 3.5 -- a real live run-resource-verify.sh
end-to-end run: VERDICT result=PASS, closing design.md's Testing
Strategy "Discovery-path regression" check specifically). Full-repo
sweep: 181 hits, zero unaccounted, byte-identical to task 4.1's
non-Makefile hits (the only delta is the Makefile's own 8
self-referential lines, now gone since the file doesn't exist).
Independently re-verified by the task reviewer.
Meson is now the single build/test system for apiserver and xcecli.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Comment-only edits in 4 files: test_ha_bootstrap.c/test_ha_ctrlhello.c
rename their "make test-integration" pointers to reference the
regression suite (tests/regression/run.sh) directly; apiserver/xcecli
cmd/main.go drop the Makefile clause from their ldflags-injection
comments, keeping only src/tools/go_build. Zero production/test
logic changed -- verified via full controller unit-test suite rerun
(1261 tests, 0 failures) and xcecli build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.
Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Rewords deploy-ha.sh's block comment and two fallback echo lines, and
run.sh's XCE-not-found error, to stop naming the now-deleted
Makefile's build-server/build-cli targets, pointing at /xce-build
apiserver instead. Discovery/control-flow logic in both files is
unchanged (verified line-by-line: no if/elif/else lines touched);
run.sh's XCE default placeholder is untouched by design.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds a new `test('apiserver backend', ...)` in apiserver/meson.build
reusing the existing apiserver_cgo_deps-derived CGO flags (resolving
their @CURRENT_SOURCE_DIR@/@BUILD_ROOT@ placeholder tokens via
Meson's own .replace(), since this target invokes go_compiler
directly rather than through tools/go_build). Adds timeout: 300
(matching the existing controller test's convention) since a
cold-cache CGO+ASAN build exceeds Meson's 30s default.
Verified cold and warm cache; confirmed CGO_ENABLED=1 selects the
real backend/ CGO path, not the //go:build !cgo stub.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
The Nix build previously fetched Go modules from goproxy.cn and Python
build deps from PyPI during the build phase, relying on network leaking
into the sandbox. This is non-reproducible and breaks whenever the
sandbox resolver is unreachable (e.g. pc13v's loopback dnsmasq).
Go:
- Vendor all three modules (frontend/apiserver, frontend/xcecli,
autoscaler/providers/baidu/src/bcctool) and commit vendor/ trees.
- tools/go_build now builds with GOFLAGS=-mod=vendor and GOPROXY=off
instead of GOPROXY=https://goproxy.cn.
Python (xceadmin):
- Add setuptools + wheel to xce.nix nativeBuildInputs.
- meson-build.sh installs with pip --no-build-isolation --no-index
--no-deps so no PyPI access happens at build time.
xceadmin install script:
- Invoke meson-install.sh via find_program('bash') instead of relying
on its #!/usr/bin/env bash shebang, which fails in the Nix install
sandbox (no /usr/bin/env -> 'could not be run').
Cleanup:
- Drop now-unneeded cacert and impureEnvVars=[HOME] from xce.nix and
xcecli.nix (they only existed to enable the impure network fetch).
Docs:
- README build guide: drop '--option sandbox false' from nix build /
nix bundle commands, note the build is now hermetic, and document the
go mod vendor + git add vendor/ maintenance workflow.
Verified: nix build .#xcecore, .#xcecore_debug, .#xcecli, and .#xce all
succeed with the sandbox enabled.
xcecli hosts list showed NUM_RUN/NUM_SSUSP/NUM_USUSP as always 0 --
the fields existed on HostInfo and were displayed, but nothing ever
assigned them. processHostData now derives per-host slot usage from
the live job store (exec_hosts + job_status), counting occupied slots
so the numbers are comparable to MAX_SLOT.
Partial fix for gitea#46 / bug-1743: the separate "dispatch stops at
8/16" question needs the reporter's live limits.yaml/pend-reason to
confirm a usergroup/queue slot limit, not addressed here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
xcecli jobs kill/stop/resume/checkpoint failed for every job with an
empty error body. models.JobInfo.JobStatus/.ArrayIdx are plain Go
`int`, but validateJobControl/foundJob asserted them as `int64` via a
bare type assertion, which always panicked; Gin's Recovery()
middleware swallowed the panic and returned an empty 500.
Fixes gitea#45 / bug-1742.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
apply_committed_entry() (persist.c), the single seam that applies every
committed WAL entry, only ever fired the generic HOOK_DETAIL_JOB_UPDATE
for job-state changes -- HOOK_DETAIL_JOB_START/JOB_EXIT were
permanently dead code, since job.c only ever fires them at
HOOK_BEFORE_EVENT (for WAL-append purposes). This left sched_job_start_
fun/sched_job_exit_fun (sched.c) and apiserver_job_start_fun/
apiserver_job_exit_fun (apiserver.c) unreachable, and the Go apiserver's
already-wired OnJobFinished handler permanently starved of data -- a
finished job's state could stay stuck (e.g. showing RUN) in the
scheduler's own view and the apiserver never learned a job finished.
The wire format was already enriched by bug-1697/bug-1689 with the
pidinfo/exit_info data these hooks need, so this closes the gap rather
than requiring new wire work. WAL_JOB_FINISH now fires
HOOK_DETAIL_JOB_EXIT; WAL_JOB_START now fires HOOK_DETAIL_JOB_START,
gated on a call-scoped decode flag rather than the transition result
(WAL_JOB_START applies as a no-op duplicate for every real job in
production per bug-1697, so gating on the transition result would make
it dead code again). WAL_JOB_FAIL -> HOOK_DETAIL_JOB_FAIL has the same
gap and is left as a follow-up.
Also includes this investigation's supporting changes: a DEBUG3 log-
retention bump (main.c, both controller and scheduler) so race-condition
evidence survives a live burst without rotating out, and DEBUG3-gated
diagnostic trace logging at the two points a job-update notification
can fire or silently drop (job.c, sched.c).
Live-verified end to end on a real deployed cluster: scheduler-side
sched_job_state_change now fires for RUN and EXIT states, and the
apiserver's OnJobFinished -- previously never invoked in production --
now correctly updates job status on completion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
FilterRows routed ANY condition on job_status/host_status through
MatchesBitmaskCondition based purely on the field name, regardless of
the condition's actual operator. MatchesBitmaskCondition only
implements OpEQ/OpNE (the only two shapes buildIntCondition's in/notin
bitmask branch ever produces) and silently returns false for anything
else -- so xcecli's `-a` flag, which builds `{job_status:notnull:}`
(Operator: OpNOTNULL), matched zero rows unconditionally. `xcecli jobs
list -a` showed "no job found" even when jobs genuinely existed and
were correctly queryable through every other view.
FilterRows now requires the operator to actually be OpEQ/OpNE before
taking the bitmask path; any other operator on these fields correctly
falls through to the generic MatchesCondition evaluator, which already
implements OpNOTNULL/OpISNULL correctly -- no changes needed to either
evaluator itself, only the dispatch condition.
New query_test.go (no test file existed for this package before):
TestFilterRows_JobStatusNotNull_MatchesRealRows reproduces the bug
(independently confirmed RED: 0 of 3 rows matched) and
TestFilterRows_JobStatusBitmaskEQ_StillWorks locks in that the actual
bitmask shape used by every other status flag is unaffected. Live-
verified: `xcecli jobs list -a` now correctly shows jobs across every
status, immediately, no restart needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
