Help of XCE
===========

This is the entrypoint of xce tool. Currently, we have following sub-commands
available:

- **controller** — start xce controller daemon
- **worker** — start xcecd worker daemon
- **scheduler** — start xce scheduler daemon
- **apiserver** — start xce REST API server
- **shell** — interactive bash shell for troubleshooting

If you run xce without any parameter or with a parameter not listed above, it
will show this help screen.

Note: `xcecli` is not a sub-command of `xce` — it is a separate standalone
Go binary invoked directly. See the [xcecli](#xcecli) section below.

## controller

Start the xce controller daemon with the given parameters after the
`controller` sub-command.

Example:
```
xce controller --workdir /var/lib/xce/controller --myname ctrl01 --peers ctrl01:9990,ctrl02:9990
```

## scheduler

Start the xce scheduler daemon with the given parameters after the
`scheduler` sub-command.

## worker

Start xcecd with the given parameters after the `worker` sub-command.

## apiserver

Start the xce REST API server with the given parameters after the
`apiserver` sub-command. The API server provides HTTP endpoints for job
submission, queue management, and cluster queries, backed by JWT
authentication and RBAC authorization.

## xcecli

The xce CLI (`src/frontend/xcecli/`) provides command-line access to the
cluster. Common operations:

- `xcecli jobs submit <jobspec>` — submit a job
- `xcecli jobs list` — list jobs
- `xcecli queues list` — show queue state

Built as a standalone Go binary; communicates with the API server over
its REST interface.

## Killing and stopping jobs

Job control follows LSF's TERMINATE and SUSPEND actions:

- **Kill** (`xcecli jobs kill <id>`, no signal given): SIGINT, then SIGTERM,
  then SIGKILL, one interval apart, each only while the job is still
  running, so the job can catch SIGINT/SIGTERM and clean up.
- **Stop** (`xcecli jobs stop <id>`): SIGTSTP first, so the job can catch
  it, prepare and stop itself; SIGSTOP follows after the interval if it has
  not stopped. **Resume** sends SIGCONT and cancels a stop still in progress.
- **Explicit signal**: that signal only, once (`xcecli jobs kill -s
  <name|number> <id>`, e.g. `-s TERM`, `-s SIGUSR1`, `-s 9`; REST API:
  `signal` in the control request, kill action only).
- **Force kill**: SIGKILL at once (`xcecli jobs kill --force <id>`; REST
  API: `force_kill: true`).

`--signal` and `--force` work with job IDs and with the filters (`-J`, `-q`,
`-u`, `-a`), and cannot be used together. `stop` and `resume` take no
signal.

The interval is the API server's `job.terminate_interval` setting
(`config/app.yaml`, seconds; default 10 when unset or 0, as LSF's
`JOB_TERMINATE_INTERVAL`; negative refuses startup).

## Job arrays

A job array is a set of identical jobs (elements) submitted at once. XCE
supports a subset of the LSF array syntax.

### Supported

- **Submitting.** `xcecli jobs submit -J 'name[1-N]' <command>` submits an
  array of N elements, numbered 1 through N. `-J 'name[1]'` submits a
  one-element array. Element numbers always start at 1.
- **Array size limit.** N may be at most the API server's
  `max_array_size` setting (`config/app.yaml`; default 1000 when unset or 0,
  never above 32767 — larger values are clamped to 32767 with a startup
  warning). A larger N is rejected with a message stating the effective limit.
- **Array name.** The part before `[` is 1 to 4094 characters, each an ASCII
  letter, digit, `.`, `_` or `-`. No whitespace is allowed anywhere in the
  expression.
- **Submit result.** Submission reports the array identity, without an
  element number, and the number of elements created, e.g.
  `Job <array-id> is submitted to queue <normal> (N elements).` The REST
  response carries the same identity and an `array_size` field.
- **Referring to elements.** Each element is shown and addressed as
  `<array-id>[k]`, in listings, job detail output, control command results and
  in log entries that print a job identity in its written form (some log
  entries still print the raw job number). Element numbers in a selection are
  at most 32767.
  - Listing (`xcecli jobs list`) and control commands (`kill`, `stop`,
    `resume`, `checkpoint`) accept one element, `<array-id>[k]`.
  - Control commands (`kill`, `stop`, `resume`, ...) also accept a range,
    `<array-id>[a-b]` (`a` equal to `b` allowed). Queries accept a range only
    by job name, e.g. `-J 'name[a-b]'`.
  - `-J 'name[k]'` selects element k of every array named `name`.
  - An identity without `[...]` selects every element of that array (and any
    ordinary job with that identity), as before.
  - In listing and control commands, a bare element job number (for example
    the value of `XCE_JOBID`) is rejected: name the element as
    `<array-id>[k]`.
  - `xcecli jobs peek` and `xcecli jobs restart` take one job: a job ID,
    or one element as `<array-id>[k]` (not a range). `peek` also accepts
    an element's own job number (`XCE_JOBID`). Restarting a bare array
    identity is refused: restart one element at a time.
- **Element number in the job's environment.** A running element finds its
  own number in `XCE_JOBINDEX` (1..N). An ordinary job gets
  `XCE_JOBINDEX=0`. A user-supplied `XCE_JOBINDEX` is overridden, with a
  warning in the worker log. `XCE_JOBID` is the element's own job number in
  its written form (`cluster-apiserver-timestamp-sequence`, the same form
  listings and control commands accept).
- **Test harness.** `xce_test jobsub -a N` submits elements 1 through N
  (2 <= N <= 32767, regardless of `max_array_size`; `-a 1` submits an
  ordinary job). `$jobid` is then the array identity and `$jobid[k]` names
  element k (for example `expect-job $jobid[3]`).

### LSF array features not supported

Each of these is rejected at submission with a message naming the problem; no
job or element is created.

- **Starting numbers other than 1**, e.g. `name[0-9]` ("element numbers start
  at 1") or `name[1001-2000]` / `name[5]` ("only name[1-N] and name[1] are
  currently supported").
- **Index lists**, e.g. `name[1-5,10]` or `name[1,3,5]` ("only a single range
  name[1-N] is supported").
- **Index steps**, e.g. `name[1-10:2]` (same message).
- **`%N` running limits**, e.g. `name[1-100]%5` ("running limits are not
  supported").
- **Interactive arrays**: an interactive job (`-I`) whose name contains `[`
  or `]` is rejected ("interactive jobs cannot be arrays").

These LSF features are not available either:

- **Per-element `%I`/`%J` file-name substitution.** `%I` and `%J` in file
  names (`-i`, `-o`, `-e`, output directory, working directory) are replaced
  once, at submission, with the first element's values: `%I` becomes `1` and
  `%J` becomes `<array-id>[1]` for every element. All elements therefore write
  to the same files unless the job script chooses its own file names from
  `XCE_JOBINDEX`.
- **`LSB_JOBINDEX` / `LSB_JOBID` variables** are not set. Scripts read
  `XCE_JOBINDEX` (element number) and `XCE_JOBID` (job number) instead.

## Resource requirement (resreq)

A job's resource request is given at submission with
`xcecli jobs submit -R "<request>"` (REST: `resreq`). The grammar accepts
`select[...]`, `rusage[...]`, `span[...]`, `same[...]` and `affinity[...]`
clauses in any order, compound requests (`N*{...} + M*{...}`) and
alternatives (`{...} || {...}`). A job with no request (or an empty one)
is scheduled as `type == any` (any host) — the API layer does not apply a
default; the "default: type == local" note in the API model is historical.

What the scheduler actually enforces today:

- **`select[<expression>]`** — enforced. The expression is evaluated
  against each candidate host's registered resources (`resources.yaml`:
  `slots`, `mem`, load indexes, host attributes `type`/`model`/... and any
  static resources). Workers report `type` as `LINUX64` and `model` as
  `Intel64`.
  - An unknown resource name in the expression is **tolerated, not
    rejected**: the select is dropped and the job is scheduled without it,
    with a warning in the scheduler log. Everything after the dropped
    select in the same request (e.g. a trailing `rusage[...]`) is dropped
    with it. A malformed select (syntax error, type mismatch) rejects the
    whole request at submission or scheduling.
- **`rusage[<res>=<value>[:duration=<d>[:decay=<0|1>]] ...]`** — the
  value is enforced as a per-task request (multiplied by the job's slot
  count; `mem` also feeds memory limits and accounting). The optional
  `per` suffix (`value/task`, `value/job`, `value/host`) is parsed and
  kept, and `value/job` / `value/host` are honored when computing
  job-wide totals, but the allocation request itself is always built
  **per task**. `duration` and `decay` are parsed and stored but **not
  consumed** — no historical-usage decay is implemented (see gaps.md).
- **`span[...]`, `same[...]`, `affinity[...]`** — parsed for syntax and
  **dropped**: a multi-host job may be split across hosts by even
  splitting only, with no placement control (see gaps.md #15).
- **Queue-level `res_req` / `gpu_req`** (`queues.yaml`) — the queue's
  `res_req` string is carried to the scheduler's queue object but is
  **never merged into job allocation**; jobs get only their own request.
  `gpu_req` has no wire field at all.
- **Ordering** — requests are evaluated first-fit in submission order
  within a queue; there is no `order[...]` clause and no load-based
  ordering.

## shell

An interactive bash shell used for troubleshooting and inspection. Not
intended for normal cluster operation.