type=array_stats (xce jobs list -A) read store.ArrayJobs, which has no
production writer, so it always answered zero rows. Stats are now derived
at query time by scanning store.Jobs and aggregating per array base ID
(controllers/arraystats.go), mirroring hosts.go's computeHostSlotUsage
precedent: counters can never drift from the element rows.
- GetJobsInfo/GetJobByIDs array_stats branches run the derived rows
through the same ExecuteQuery filter/sort/paginate pipeline as store
queries; unique_job_id rides in the field list so processJobData can
annotate job_id_human (SelectFields drops unlisted fields).
- processJobData: new rowInt64 helper (uint64 bit-preserved + float64/
json.Number tolerant) replaces brittle bare type assertions for
unique_job_id/array_idx reads; dead code removed (StackLimit
wrong-source copy, unreachable index error).
- QueryData(TableArrayJob) now returns a guard error instead of always
answering zero rows from the never-written store.
- xcecli: ArrayJobStatsData got UnmarshalJSON via UnmarshalOrderMapRow —
the server's OrderMap wrapper ({Fields,Data}) decoded to all-zero
fields, which is why table mode showed 0/- rows against live arrays.
- analysis.md CL5 marked fixed with the real root cause.
Cluster-verified on t1/t2/t3: submit→PEND→RUN→DONE per element, kill
path, job_id filter, and per-status counters all live; all daemons
healthy. Unit suites: apiserver + xcecli green (CGO_ENABLED=0).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Render -X jobs' windows on the user's own X server with zero client
install, mirroring the ssh -X experience: the job command starts with
DISPLAY pointed at a per-job TCP listener on the worker, and the user's
xcecli relays that listener's byte stream to their local X server.
- xcecli: new -X/--x11 flag (interactive=2 + exdata "x11":1). After
submit it fetches the per-job MIT-MAGIC-COOKIE-1, registers it in the
user's Xauthority, then bridges one xattach websocket to the local X
server address parsed from $DISPLAY (unix socket or TCP). The cookie
entry is removed when the session ends. Pre-flight checks report
missing DISPLAY/xauth before submit.
- apiserver: GET /xce/v1/jobs/:id/xattach relays websocket binary frames
to the worker's forwarding port (raw TCP, no ttyd). Close codes match
the ttyd attach (4000 unreachable, 4001 ended). The job-finished event
ends the relay. Teardown closes both hops as soon as either pump
exits, so a client disconnect propagates instead of deadlocking in a
blocking read.
- worker: XJOB jobs with exdata x11=1 skip VNC, listen on 6100-6199,
generate a per-job xauth file (/tmp/job_<id>_xauth, worker-side since
the worker has no session), export DISPLAY/XAUTHORITY to the command,
and bridge the app connection (loopback) with the relay connection
(non-loopback; roles classified by source address, not arrival order).
A 60 s no-client watchdog reaps jobs nothing ever attached to; the
v1 single-connection boundary refuses a second X connection loudly.
t1-verified per the plan matrix: window rendering (xclock/xeyes on
Xvfb), normal end -> DONE + xauth removed, no-client 60 s reap (137),
kill -> immediate teardown with no residue, -I/-I --browse regression,
daemon health, missing-DISPLAY error, second-connection refusal.
(--xjob VNC on t1 fails for lack of a vncserver binary; pre-existing.)
Co-Authored-By: Claude Code <noreply@anthropic.com>
An attach job (xcecli -I without --browse) that no client ever connects
to sat in RUN forever: ttyd only starts the job command on the first
websocket client, so with no client the command never runs, ttyd never
exits, and the job holds its ttyd, port and per-job worker indefinitely.
New exdata key "attach":1, set by the CLI for TTY jobs that are not
--browse and carried through the apiserver's extendDataMap to the
worker's transfer_info->attach_mode. For such jobs the worker starts a
1 s watchdog when the job process starts:
- no client attached within 60 s -> reap ttyd (SIGKILL to the monitored
tree), job EXITs with status message "No client attached within 60 s"
- client came and the command ran and exited -> give ttyd 5 s to stop
on its own (-Q), then reap: "Command exited; ttyd did not stop on its
own"
The reap lands in the regular process_exit_callback path (JOB_EXITED,
status reason, post-exec, worker exit), so kill and cleanup behave like
any other job termination.
Field-critical detail: uv_proc_get_pid returns the "/bin/sh -c ttyd ..."
wrapper pid, and sh does not reliably exec -- on the field ttyd runs as
a child of the wrapper. Judging the wrapper's children would see ttyd
itself as "the command has started" and never reap. The watchdog
resolves the real ttyd pid first (/proc/<pid>/comm walk down through
the sh wrapper, capped depth) and judges client activity by the real
ttyd's children; killing still goes through the monitored wrapper tree
so both topologies (exec'd or forked) are covered.
Web-viewed sessions (--browse / DISPLAY) are untouched: no attach
exdata, no watchdog -- ttyd stays until a browser connects or the job
is killed, as before.
Attach failure on the CLI side prints that the job will be cleaned up
by the worker if no client attaches within 60 s.
Tests: xcecli (attach flag matrix), apiserver controllers (attach
exdata end-to-end in ExtendData), worker native suites; live t1
cluster: normal attach completes DONE with no residue, unattached job
EXITs at ~60 s with the reason in the worker log and port released,
Ctrl-C-then-wait completes when the command ends, --browse job stays
RUN past 60 s and kills cleanly.
Co-Authored-By: Claude Code <noreply@anthropic.com>
RestartJobByID rebuilt the new job from the old job's row:
- it accepted asked_hosts and copied asked hosts, I/O paths and out_dir,
which no job gets (refused at submit since bug-1936); restart now
refuses asked_hosts with 400 and no longer carries those over;
- it read the JSONB columns chkpnt_info/extend_data as strings, but rows
keep models.Jsonb, so env/limits were lost and a checkpointed job never
restarted from its checkpoint; new rowJSON() reads both;
- it read "res_req" (the column is "resreq"), losing the resource
requirement, and never copied exec_home.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: CreateJob accepted in_file, out_file, err_file, out_dir,
asked_hosts, begin_time, term_time and transfer_files and returned
success, but neither the controller's submit data/start info nor the
worker carries any of them (the worker never stages transferred files),
so they were silently dropped. Exec home had no submit field at all, and
a relative cwd would have been resolved against the worker unit's
directory.
Fix: new preCheckSubmitOptions() runs right after binding, before file
uploads and ID allocation, and answers 400 naming every unsupported
option that is set; cwd and the new exec_home field must be absolute and
fit the worker's job_assignment_t buffers (1023 bytes). exec_home goes
to JobInfo.ExecHome and from there (submitStartInfo) to the start info's
exechome. num_files already reaches exdata as nofile_limit; a handler
test now pins it. The array %J/%I submit-time substitution test (9.4)
checks cwd instead of out_file/err_file, which are now refused. Swagger
regenerated with `swag init -g cmd/main.go -o docs`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since the controller rejects submissions to closed or unknown queues
(E_XCE_QUEUE_CLOSED, E_XCE_BAD_QUEUE), the apiserver answered them as a 500
"xcecli_jobsubmit failed: -10016". SubmitJob now maps them to
backend.ErrQueueClosed / ErrQueueInvalid (numbers checked against errnox.h
at compile time), and the submit and restart handlers answer 400 naming
the queue, as the apiserver's own pre-check does. This covers a queue
closed between that check and the controller's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: preCheckJobRequest (controllers/jobs.go) ran the reserved-name,
existence and closed checks only when a queue name was given; an empty
name was set to "normal" and skipped them, so jobs were accepted into a
closed default queue.
Fix: default the name first, then run the same checks for every
submission. Two existing tests now seed the default queue. (The controller
side of bug-1960 is fixed separately.)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: modelJobToSubmit() filled only cmdline, execuser, launch_type and
pre/post-exec of job_startinfo_t, so the extended data the submit handler
builds (env, *_limit incl. -W run limit, file transfers, interactive mode),
the cwd and the exec home never reached the controller or the worker.
Fix: map the start info in a pure submitStartInfo() (exdata from
ExtendData, execcwd from ExecCWD or CWD, exechome from ExecHome) that
modelJobToSubmit copies into the C struct; decode exdata back in
cJobToModel/cJobArrayToModels so the store keeps ExtendData and CWD.
Reject extend_data the worker's parse_exdata() cannot take (not a JSON
object, over EXDATA_LENGTH-1 bytes) and drop an empty transfer list
instead of sending a null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1934: every job-permission check read a single scope and compared it
only with "self", so a job:view or job:control grant scoped to self_ug acted
like "all" (list, get by ID, control), and self + self_ug collapsed to self.
userJobScope() now collects all of a user's grants of a job permission
before deduplication and adds them up (all; self; self_ug = the user and
everyone sharing a user group). The job list appends a submit_user filter
for narrower scopes, get-by-ID keeps covered rows in a new slice (removing
from the ranged slice skipped rows) and answers 403, both control paths
and job output check the owner against the scope. User groups now reach
the apiserver: the config push converts cluster_config_t.ugroups and
OnUserGroupsUpdated() replaces the new store (nested groups, all,
exclusions). The default basic_user now also sees its user-group mates'
jobs, as its configuration says.
bug-1935: project_name and license_project are accepted job filters, so
xcecli jobs list -P works.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- apiserver peek (parseJobIDParam): <array-id>[k] now resolves to the
element; it used to drop [k] and look up the array identity itself
(and panicked in parseArrayName on some inputs). A job number still
names that job exactly (an element by its own XCE_JOBID); ranges and
invalid selections are rejected.
- apiserver restart: the decimal-only pre-check (isValidJobId, now
unused and removed) rejected the written form c-a-ts-seq[k];
ParseIdentitySelection, already used for the lookup, validates it.
- xcecli jobs restart converts its argument with ToAPIJobID like the
control commands, so written IDs work and invalid ones fail locally.
- docs: the peek/restart caveat in the Job arrays section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
RestartJobByID looked the job up with a job_id CompCondition, which the
store never evaluates, and restarted jqueryReply[0] -- the first stored
job -- for any ID, including unknown ones. It now parses the ID with
ParseIdentitySelection, fetches the rows by key and matches them with
foundJob, as ControlJobByID does: an ordinary job or <array-id>[k] is
restarted, an unknown job or element is not found, and a bare array
identity (several elements) is refused. Restart goes through the
generateJobID/submitJob seams so the handler fixture can test it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follows page_token with limit=3 over the handler fixture. Before
efff6b65 the token's float-rounded array-element ID matched no row and
every page restarted at row 0 (the original repeat-the-first-page
report); it now pages through every row exactly once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Filter-based ControlJob validates a bracketed -J value before fetching
(400 with the parser's message; unbracketed names, including commands
with spaces, unchanged). FetchJobList evaluates bracketed job_name
entries through the selection predicates instead of unevaluated
compound conditions, so -J 'foo[1]' acts on element 1 of every array
named foo and never on ordinary jobs of that name. The most-recent-job
rule applies only without an element number and without all-jobs, and
treats the element of a one-element array like an ordinary job
(isSingleElementArray, a documented approximation). The old
index-section parser is deleted. Completes bug-1898 on the owned paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ControlJobByID parses every selection with ParseIdentitySelection before
any store access (first rejection: 400, nothing dispatched), collects
the jobs to act on de-duplicated by unique_job_id plus one not-found
result per missing number per selection (carrying the would-be element
ID), validates each job once after the loop and dispatches each
accepted job once. foundJob gains an explicit-index flag so
<array-id>[0] matches only element 0 and never falls back to the whole
array or an ordinary job. Fixes bug-1899 (earlier selections were
re-dispatched for every later one).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GetJobsInfo validates every filter entry in request order before any
store access (including the exec_hosts lookup) and GetJobByIDs every key
before querying; the first rejection returns 400 with the parser's
message. Bracketed job_id/job_name entries are evaluated as row
predicates before sorting and pagination, so job_id:eq:X[2] lists only
element 2, GET /jobs/X[1-3] lists elements 1-3 and <ordinary-id>[0]
lists nothing. The unevaluated compound conditions and the old index
parsing are gone from both handlers. Fixes bug-1898 on the query paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
preCheckJobRequest returns (n, isArray, err): a bracketed job name is
parsed by ParseSubmitIndexExpr first, so name[1-N] and name[1] create
arrays (one-element arrays included) allocated with
generateArrayJobID, stored under the bare name; expression errors are
answered 400 with their specific message and create nothing; names
without brackets take the unchanged createArrayName path. The response
names the array (jobid = array ID, new array_size, "(n elements)"),
submit-time %I/%J use the first element, and the submit and dispatch
logs identify arrays by array ID. Fixes bug-1896 for REST and xcecli.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CreateJob's ID generation and submission and both control dispatch
sites now go through package variables (generateJobID,
generateArrayJobID, submitJob, controlJob) that default to the real
backend/services functions, so production behaviour is unchanged. A new
non-cgo handler test file provides a reusable fixture: it saves and
restores every global it touches, sets up auth/RBAC, seeds jobs and
their queues, and records each dispatched job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
