A-batch admin-plane fixes: rusage unit name, XCE_JOBID identity, select WARN, CLI env, docs

rusage collection (merge cb0c7fe9 regression, root cause of 0 counters):
- collect_job_rusages builds the systemd unit name with
  make_job_sd_unit_name (four-segment fmt_jobid_ex form) instead of a bare
  %lu of the packed uint64, so the D-Bus property lookups hit the real
  per-job unit again. E2E on t1: DONE job reports max_mem/avg_mem/cpu_time
  non-zero (previously all 0).

XCE_JOBID written identity:
- exdata sets XCE_JOBID via fmt_jobid (four-segment
  cluster-apiserver-timestamp-seq) instead of a bare packed decimal;
  XCE_JOBINDEX handling unchanged.
- xrun parses XCE_JOBID with parse_jobid (accepts the four-segment form);
  a bare decimal no longer round-trips through the written identity.
- E2E on t1: job sees XCE_JOBID=<submit id>; xrun remote-start round-trip
  to another host matches the job.

select WARN (silent degradation):
- parse_resreq_primary WARNs when a select references an unregistered
  resource ("...references an unregistered resource -- dropping the select
  and skipping any rusage[] clause of this primary") instead of failing
  silently; scheduler log shows the WARN on t1 and the job schedules
  without the constraint.

CLI env handling (unit-tested; subEnvVars is not flag-bound so this is
not reachable from the production CLI):
- getEnvironmentVariables keeps B=b=c (split at first '=') and warns +
  drops a bare key or an '=value' fragment instead of silently mis-parsing.

Docs:
- src/docs/xce.md: document the dead-clause semantics (select/rusage
  primary drop behavior) matching the WARN.

bug-1972 (test-only UAF): test_job_live_gc asserted job destruction by
reading refcount from the freed job; rewrite the assertion to observe
destruction without touching freed memory. controller_unit_test is green
again (1354 tests).

Regenerate 3 repl_trace_baseline fixtures whose baselines shifted with the
select-WARN logging; replay tests green.

Spec naming: external-license-resources design.md/brief.md fix the
self-referential daemon-name note (old drafts said xceextresd/extresd).

Verified on t1: rusage_burn job (counters non-zero in DONE), echo_env job
(four-segment XCE_JOBID), select[nosuchres] job (WARN + schedules),
xrun t2 remote-start round-trip, CLI env unit tests, xcelib 762 /
controller 1354 / worker exdata 8+11 / sched repl_trace 4+14+4+10+17
tests green.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-26 14:02:53 +00:00
parent 68eabd8a4a
commit f780a1660a
17 changed files with 329 additions and 44 deletions

View file

@ -122,8 +122,9 @@ supports a subset of the LSF array syntax.
- **Element number in the job's environment.** A running element finds its
own number in `XCE_JOBINDEX` (1..N). An ordinary job gets
`XCE_JOBINDEX=0`. A user-supplied `XCE_JOBINDEX` is overridden, with a
warning in the worker log. `XCE_JOBID` is the element's own job number, as
before.
warning in the worker log. `XCE_JOBID` is the element's own job number in
its written form (`cluster-apiserver-timestamp-sequence`, the same form
listings and control commands accept).
- **Test harness.** `xce_test jobsub -a N` submits elements 1 through N
(2 <= N <= 32767, regardless of `max_array_size`; `-a 1` submits an
ordinary job). `$jobid` is then the array identity and `$jobid[k]` names
@ -156,6 +157,48 @@ These LSF features are not available either:
- **`LSB_JOBINDEX` / `LSB_JOBID` variables** are not set. Scripts read
`XCE_JOBINDEX` (element number) and `XCE_JOBID` (job number) instead.
## Resource requirement (resreq)
A job's resource request is given at submission with
`xcecli jobs submit -R "<request>"` (REST: `resreq`). The grammar accepts
`select[...]`, `rusage[...]`, `span[...]`, `same[...]` and `affinity[...]`
clauses in any order, compound requests (`N*{...} + M*{...}`) and
alternatives (`{...} || {...}`). A job with no request (or an empty one)
is scheduled as `type == any` (any host) — the API layer does not apply a
default; the "default: type == local" note in the API model is historical.
What the scheduler actually enforces today:
- **`select[<expression>]`** — enforced. The expression is evaluated
against each candidate host's registered resources (`resources.yaml`:
`slots`, `mem`, load indexes, host attributes `type`/`model`/... and any
static resources). Workers report `type` as `LINUX64` and `model` as
`Intel64`.
- An unknown resource name in the expression is **tolerated, not
rejected**: the select is dropped and the job is scheduled without it,
with a warning in the scheduler log. Everything after the dropped
select in the same request (e.g. a trailing `rusage[...]`) is dropped
with it. A malformed select (syntax error, type mismatch) rejects the
whole request at submission or scheduling.
- **`rusage[<res>=<value>[:duration=<d>[:decay=<0|1>]] ...]`** — the
value is enforced as a per-task request (multiplied by the job's slot
count; `mem` also feeds memory limits and accounting). The optional
`per` suffix (`value/task`, `value/job`, `value/host`) is parsed and
kept, and `value/job` / `value/host` are honored when computing
job-wide totals, but the allocation request itself is always built
**per task**. `duration` and `decay` are parsed and stored but **not
consumed** — no historical-usage decay is implemented (see gaps.md).
- **`span[...]`, `same[...]`, `affinity[...]`** — parsed for syntax and
**dropped**: a multi-host job may be split across hosts by even
splitting only, with no placement control (see gaps.md #15).
- **Queue-level `res_req` / `gpu_req`** (`queues.yaml`) — the queue's
`res_req` string is carried to the scheduler's queue object but is
**never merged into job allocation**; jobs get only their own request.
`gpu_req` has no wire field at all.
- **Ordering** — requests are evaluated first-fit in submission order
within a queue; there is no `order[...]` clause and no load-based
ordering.
## shell
An interactive bash shell used for troubleshooting and inspection. Not