myxce/src/docs/xce.md

Help of XCE

This is the entrypoint of xce tool. Currently, we have following sub-commands available:

• controller — start xce controller daemon
• worker — start xcecd worker daemon
• scheduler — start xce scheduler daemon
• apiserver — start xce REST API server
• shell — interactive bash shell for troubleshooting

If you run xce without any parameter or with a parameter not listed above, it will show this help screen.

Note: xcecli is not a sub-command of xce — it is a separate standalone Go binary invoked directly. See the xcecli section below.

controller

Start the xce controller daemon with the given parameters after the controller sub-command.

Example:

xce controller --workdir /var/lib/xce/controller --myname ctrl01 --peers ctrl01:9990,ctrl02:9990

scheduler

Start the xce scheduler daemon with the given parameters after the scheduler sub-command.

worker

Start xcecd with the given parameters after the worker sub-command.

apiserver

Start the xce REST API server with the given parameters after the apiserver sub-command. The API server provides HTTP endpoints for job submission, queue management, and cluster queries, backed by JWT authentication and RBAC authorization.

xcecli

The xce CLI (src/frontend/xcecli/) provides command-line access to the cluster. Common operations:

• xcecli jobs submit <jobspec> — submit a job
• xcecli jobs list — list jobs
• xcecli queues list — show queue state

Built as a standalone Go binary; communicates with the API server over its REST interface.

Killing and stopping jobs

Job control follows LSF's TERMINATE and SUSPEND actions:

• Kill (xcecli jobs kill <id>, no signal given): SIGINT, then SIGTERM, then SIGKILL, one interval apart, each only while the job is still running, so the job can catch SIGINT/SIGTERM and clean up.
• Stop (xcecli jobs stop <id>): SIGTSTP first, so the job can catch it, prepare and stop itself; SIGSTOP follows after the interval if it has not stopped. Resume sends SIGCONT and cancels a stop still in progress.
• Explicit signal: that signal only, once (xcecli jobs kill -s <name|number> <id>, e.g. -s TERM, -s SIGUSR1, -s 9; REST API: signal in the control request, kill action only).
• Force kill: SIGKILL at once (xcecli jobs kill --force <id>; REST API: force_kill: true).

--signal and --force work with job IDs and with the filters (-J, -q, -u, -a), and cannot be used together. stop and resume take no signal.

The interval is the API server's job.terminate_interval setting (config/app.yaml, seconds; default 10 when unset or 0, as LSF's JOB_TERMINATE_INTERVAL; negative refuses startup).

Job arrays

A job array is a set of identical jobs (elements) submitted at once. XCE supports a subset of the LSF array syntax.

Supported

• Submitting. xcecli jobs submit -J 'name[1-N]' <command> submits an array of N elements, numbered 1 through N. -J 'name[1]' submits a one-element array. Element numbers always start at 1.
• Array size limit. N may be at most the API server's max_array_size setting (config/app.yaml; default 1000 when unset or 0, never above 32767 — larger values are clamped to 32767 with a startup warning). A larger N is rejected with a message stating the effective limit.
• Array name. The part before [ is 1 to 4094 characters, each an ASCII letter, digit, ., _ or -. No whitespace is allowed anywhere in the expression.
• Submit result. Submission reports the array identity, without an element number, and the number of elements created, e.g. Job <array-id> is submitted to queue <normal> (N elements). The REST response carries the same identity and an array_size field.
• Referring to elements. Each element is shown and addressed as <array-id>[k], in listings, job detail output, control command results and in log entries that print a job identity in its written form (some log entries still print the raw job number). Element numbers in a selection are at most 32767.
  • Listing (xcecli jobs list) and control commands (kill, stop, resume, checkpoint) accept one element, <array-id>[k].
  • Control commands (kill, stop, resume, ...) also accept a range, <array-id>[a-b] (a equal to b allowed). Queries accept a range only by job name, e.g. -J 'name[a-b]'.
  • -J 'name[k]' selects element k of every array named name.
  • An identity without [...] selects every element of that array (and any ordinary job with that identity), as before.
  • In listing and control commands, a bare element job number (for example the value of XCE_JOBID) is rejected: name the element as <array-id>[k].
  • xcecli jobs peek and xcecli jobs restart take one job: a job ID, or one element as <array-id>[k] (not a range). peek also accepts an element's own job number (XCE_JOBID). Restarting a bare array identity is refused: restart one element at a time.
• Element number in the job's environment. A running element finds its own number in XCE_JOBINDEX (1..N). An ordinary job gets XCE_JOBINDEX=0. A user-supplied XCE_JOBINDEX is overridden, with a warning in the worker log. XCE_JOBID is the element's own job number in its written form (cluster-apiserver-timestamp-sequence, the same form listings and control commands accept).
• Test harness. xce_test jobsub -a N submits elements 1 through N (2 <= N <= 32767, regardless of max_array_size; -a 1 submits an ordinary job). $jobid is then the array identity and $jobid[k] names element k (for example expect-job $jobid[3]).

LSF array features not supported

Each of these is rejected at submission with a message naming the problem; no job or element is created.

• Starting numbers other than 1, e.g. name[0-9] ("element numbers start at 1") or name[1001-2000] / name[5] ("only name[1-N] and name[1] are currently supported").
• Index lists, e.g. name[1-5,10] or name[1,3,5] ("only a single range name[1-N] is supported").
• Index steps, e.g. name[1-10:2] (same message).
• %N running limits, e.g. name[1-100]%5 ("running limits are not supported").
• Interactive arrays: an interactive job (-I) whose name contains [ or ] is rejected ("interactive jobs cannot be arrays").

These LSF features are not available either:

• Per-element %I/%J file-name substitution. %I and %J in file names (-i, -o, -e, output directory, working directory) are replaced once, at submission, with the first element's values: %I becomes 1 and %J becomes <array-id>[1] for every element. All elements therefore write to the same files unless the job script chooses its own file names from XCE_JOBINDEX.
• LSB_JOBINDEX / LSB_JOBID variables are not set. Scripts read XCE_JOBINDEX (element number) and XCE_JOBID (job number) instead.

Resource requirement (resreq)

A job's resource request is given at submission with xcecli jobs submit -R "<request>" (REST: resreq). The grammar accepts select[...], rusage[...], span[...], same[...] and affinity[...] clauses in any order, compound requests (N*{...} + M*{...}) and alternatives ({...} || {...}). A job with no request (or an empty one) is scheduled as type == any (any host) — the API layer does not apply a default; the "default: type == local" note in the API model is historical.

What the scheduler actually enforces today:

• select[<expression>] — enforced. The expression is evaluated against each candidate host's registered resources (resources.yaml: slots, mem, load indexes, host attributes type/model/... and any static resources). Workers report type as LINUX64 and model as Intel64.
  • An unknown resource name in the expression is tolerated, not rejected: the select is dropped and the job is scheduled without it, with a warning in the scheduler log. Everything after the dropped select in the same request (e.g. a trailing rusage[...]) is dropped with it. A malformed select (syntax error, type mismatch) rejects the whole request at submission or scheduling.
• rusage[<res>=<value>[:duration=<d>[:decay=<0|1>]] ...] — the value is enforced as a per-task request (multiplied by the job's slot count; mem also feeds memory limits and accounting). The optional per suffix (value/task, value/job, value/host) is parsed and kept, and value/job / value/host are honored when computing job-wide totals, but the allocation request itself is always built per task. duration and decay are parsed and stored but not consumed — no historical-usage decay is implemented (see gaps.md).
• span[...], same[...], affinity[...] — parsed for syntax and dropped: a multi-host job may be split across hosts by even splitting only, with no placement control (see gaps.md #15).
• Queue-level res_req / gpu_req (queues.yaml) — the queue's res_req string is carried to the scheduler's queue object but is never merged into job allocation; jobs get only their own request. gpu_req has no wire field at all.
• Ordering — requests are evaluated first-fit in submission order within a queue; there is no order[...] clause and no load-based ordering.

shell

An interactive bash shell used for troubleshooting and inspection. Not intended for normal cluster operation.