merge: sync Rongsong's scheduler-framework (c5b172c7..3442a57d, 781 commits)

Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1, scheduler-agent Phase 2, bug-924 owner-pid shim). Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/ (92 files dropped); .claude/ + CLAUDE.md keep ours. Key theirs-side changes absorbed: - bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed queue refuses new submissions only; pending jobs still run) -- supersedes our QUEUE_CLOSED preason on the dispatch path; sched_queue_status_reason drops that branch - bug-1846: config push now stashes and applies between scheduling rounds (sched_stash_config/sched_apply_pending_config), applied at the same main-thread round boundary that preserves bug-1513 serialization - bug-1760: sched_check_pending_rejoin exposed + xcecli_is_connection_broken public predicate - bug-1936: execuser/exechome/execcwd/exdata carried on submission (superset of our bug-1500 execuser fallback; our job-typed resolveExecUser removed as dead code) - bug-1945~1962: apiserver auth hardening (HS256 pinning, default- secret rejection, logout revocation, bcrypt, RBAC) -- our API-key branch folded into controllers.ParseTokenClaims so both coexist - bug-1736: rawRunUsage job run-usage fields in the Go model - xce_test relocated src/controller/testsuite/test.c -> src/harness/test.c; controller suite split into controller_test.c + fixtures; their clitest fixtures spawn xce_test mid-suite under `meson test -j` Ours preserved across the merge: - PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) -- gate re-ported into their relocated allocator (sched_election.c) after plugin_job_election, with attach on the allocator thread only (bug-1521 race discipline) - PEND-reason P0 main_reason semantics RECONCILED to theirs: the merge initially re-carried our first-detail-add-promotes-main_reason fallback in add_job_preason (4e2ac86), but their whole trace/verify stack treats "detail-only => main_reason 0" as a contract -- sched_trace_capture counts main_reason as its own occurrence ("Expected 2 Was 3" in test_trace_capture), the outcome renderer prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e), and the committed .replay/.sim_verify baselines encode main_reason=0 (test_repl_trace_baseline byte compare, test_trace_equivalence off-vs-traced). The fallback is dropped from the shared lib; the user-visible "overall reason" survives as an API-layer view: rawJobToModel (convert_pure.go) promotes the first detail row to main_reason_name when main_reason is 0 (main_reason VALUE marshals unchanged). add_job_preason keeps only the NULL-reasons guard - bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs - bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant (their jobCtrlUnionDiscriminant, same semantics) - xcecli typed PendingReason field (53c00834), bug-1522 reconnect resend, xcecli_force_reconnect + their is_connection_broken both exported in client.h/libclient - t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494), /xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2) - lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block - drop-audit: our bug-1504 fix covered by their bug-1680 (no action) bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the take-theirs audit's one "drop" was re-evaluated and REVERTED instead of re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under `meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as a child mid-suite; the wipe destroyed the parent's fixtures -> test_test_cli failures). Coverage instead: - mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since (testsuite_support/mkdtemp_clean.[ch]) - new case_scratch_sweep.[ch]: aggregate runners (controller_test.c, harness_test.c -- hundreds of cases per process) call case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's dirs are swept immediately in tearDown (after engine reaping), FAILED cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1 disables. One full controller_unit_test run was observed to leak 12G+ to ENOSPC without this, cascading into unrelated "Expected Non-NULL" failures - harness_unit_test_deps links the shim library as well Test reconciliations on adopt: - core-ready-gates: sched_queue_is_ready now gates only INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect -- covered end-to-end by test_queue_admin_control.c - test_worker_resync: the 4 restart-replay RUN assertions predate the bug-1509 B3 marking (8858afed) landing on the same lineage; replayed RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN); terminal DONE/EXIT assertions unchanged - convert_pure.go: our job-typed resolveExecUser deleted (superseded by their bug-1780/1936 two-arg version) - test_preason: the 3 P0-promotion assertions reverted to the detail-keeps-main-0 contract; new tests cover limit-code detail rows and explicit-main-wins-over-later-details; apiserver convert_test.go gains the detail-only name-promotion cases Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-14 05:17:03 +00:00 · 2026-10-14 05:17:03 +00:00 · 0c8607345e
commit 0c8607345e
parent 1fe56291ca 3442a57dd9
663 changed files with 139848 additions and 13614 deletions
--- a/src/docs/apiserver-backend-tasks.md
+++ b/src/docs/apiserver-backend-tasks.md
@ -67,7 +67,7 @@
  - `controllers/queues.go` — `preCheckQueueControlRequest` now returns `(bool, error)`; `ControlQueue` receives allFlag locally
  - `controllers/host_control_test.go` — 5 new tests: allFlag true/false, concurrent no-race (50 goroutines), invalid action, comment-too-long
  - `controllers/queue_control_test.go` — 5 new tests: same coverage for queues
-  - All tests passing (`make test`)
+  - All tests passing (via `/xce-build apiserver`'s test step)

 - [x] **#8 P0-2c: Register routes + RBAC for host control/select**
  - `router/routers.go` — `POST /xce/v1/hosts/control` → RBAC `host:control` (cluster_admin)
@ -117,7 +117,7 @@
 ## Final

 - [x] **#14 Build and test verification**
-  - `make test` in `src/frontend/` passes — all backend, controllers, store, utils, xcecli tests green
+  - `/xce-build apiserver`'s test step passes — all backend, controllers, store, utils, xcecli tests green
  - `ninja -C build` for C components passes — required meson reconfigure + CGO val_helpers.h fix
  - RBAC verification — `host:control` (cluster_admin) and `host:view` (all authenticated) confirmed in routes + rbac.yaml
  - Input safety — comment limit 512 chars; select expression limit 4094 chars (1024 reduction deferred to P1-2b)
--- a/src/docs/deployment.md
+++ b/src/docs/deployment.md
@ -211,16 +211,17 @@ xce_scheduler --peers "ctrl-1=controller-host:9990" --record-dir /var/log/xce/sc
 `--sim` runs the scheduler as a **standalone replay tool** against a previously recorded (or hand-built) `.replay` file — zero connection to any controller, safe to run against a production recording on a laptop:

 ```bash
-xce_scheduler --sim /path/to/session.replay --sim-speed 0        # replay as fast as possible
+xce_scheduler --sim /path/to/session.replay                      # replay the recording
 xce_scheduler --sim /path/to/session.replay --sim-verify         # replay + diff against the recorded decisions
 xce_scheduler --sim /path/to/session.replay --sim-output /tmp/resim.replay --sim-alt-events /path/to/edited-events.replay   # "what-if": substitute events, record the new decisions
 xce_scheduler --sim /path/to/session.replay --interactive        # step through it interactively (REPL)
 ```

- `--sim-speed`: `0` = as-fast-as-possible, `1` = real-time (default), `N>1` = `N`x real-time.
+- The simulator's clock moves only at recorded timestamps, never with the wall clock: it is set to the start snapshot's timestamp, then to each recorded event's timestamp and each session's recorded `beginRound`, exactly, even when that value is earlier than the current one. Between those points it does not move, so a session's recorded duration in simulation output (`--sim-output`) is 0, and a replay's decisions do not depend on host speed or load.
+- `--sim-speed` is deprecated and has no effect. It is still accepted with any value so existing command lines keep working, logs one deprecation warning, and will be removed in a future release; drop it from scripts.
 - `--sim-verify` re-derives decisions from the recorded events and asserts they match the decisions the file already recorded — the feature's core "deterministic replay" regression check, usable ad hoc against any recording.
 - `--sim-alt-events FILE` swaps in a different event stream after the initial snapshot for what-if branching; `--sim-output FILE` records the resulting decisions as a new `.replay` file.
- `--interactive` opens a REPL (`step`, `continue`, breakpoints, state inspection, snapshot capture) driving the *same* simulation run a batch `--sim` would — REPL and batch produce byte-identical decisions for the same input, only the pacing differs.
+- `--interactive` opens a REPL (`step`, `continue`, breakpoints, state inspection, snapshot capture) driving the *same* simulation run a batch `--sim` would — REPL and batch produce byte-identical decisions for the same input.
 - Simulation mode never opens a socket to any controller and never touches `--peers`/`--record-dir` — see `.kiro/specs/scheduler-core/design.md`'s Boundary Commitments for the full isolation contract.

 ## Deploying apiserver
@ -233,18 +234,18 @@ apiserver -d /opt/xce/apiserver

 | File | Purpose |
 |---|---|
-| `app.yaml` | HTTP(S) port, JWT secret/expiry, rate limits, CORS, IP whitelist, and the `backend:` block (`address`, `port`, `from`, `key`, optional `peers` for HA) pointing at the controller. |
+| `app.yaml` | HTTP(S) port, JWT secret/expiry, rate limits, CORS, IP whitelist, trusted proxies, the `job:` block (`max_array_size`; `terminate_interval`, the seconds between the signals of a plain kill or stop, default 10, see `xce.md` "Killing and stopping jobs"), and the `backend:` block (`address`, `port`, `from`, `key`, optional `peers` for HA) pointing at the controller. |
 | `auth.yaml` | Auth provider selection (`config_file` \| `pam` \| `ldap` \| `oauth2`); for `config_file`, embeds usernames/passwords/roles directly. |
 | `rbac.yaml` | Permission catalog + role→permission mappings (domains: `self`, `self_ug`, etc.). User→role binding comes from `auth.yaml`, not this file. |
 | `logger.yaml` | zap logger level/encoding/rotation. |

 Both `app.yaml` and `logger.yaml` are JSON-Schema-validated at startup (fail-fast on malformed config).

-### Build gotcha — use the Meson/Nix build, not `make build-server`
+### Build gotcha — apiserver needs real CGO flags, not a bare `go build`

-`make build-server` (`src/frontend/Makefile`) runs a bare `go build` with **no `CGO_CFLAGS`/`CGO_LDFLAGS` set**. The real controller-connection code (`backend/clib.go`) is CGO and needs those flags to find `libxce`'s headers. Depending on your environment this either fails to compile, or — if `CGO_ENABLED=0` is ambient — silently links `backend/stub.go` instead, producing a binary where every backend call returns `errNoCGO: backend requires CGO`. There is no startup warning; the HTTP server comes up fine and only fails at request time.
+A bare `go build` in `src/frontend/apiserver` (no `CGO_CFLAGS`/`CGO_LDFLAGS` set) is not a supported way to produce a working apiserver binary. The real controller-connection code (`backend/clib.go`) is CGO and needs those flags to find `libxce`'s headers. Depending on your environment this either fails to compile, or — if `CGO_ENABLED=0` is ambient — silently links `backend/stub.go` instead, producing a binary where every backend call returns `errNoCGO: backend requires CGO`. There is no startup warning; the HTTP server comes up fine and only fails at request time.

-**Use `/xce-build apiserver` (or `nix build .#xcecore`)** for anything that needs to actually talk to a controller — it computes the correct `CGO_CFLAGS`/`CGO_LDFLAGS` from pkg-config and produces `xce_apiserver` under `src/xce.build/frontend/apiserver/` (or the equivalent path for other profiles). Reserve `make build-server`/`make test` for the pure-Go unit-test loop (`CGO_ENABLED=0` is *intentional* there).
+**Use `/xce-build apiserver` (or `nix build .#xcecore`)** for anything that needs to actually talk to a controller — it computes the correct `CGO_CFLAGS`/`CGO_LDFLAGS` from pkg-config and produces `xce_apiserver` under `src/xce.build/frontend/apiserver/` (or the equivalent path for other profiles). `/xce-build apiserver` also runs the pure-Go unit-test loop (`CGO_ENABLED=0` is *intentional* there); a manual `go build`/`go test` invocation is not the canonical path.

 ### Relationship to the controller

@ -284,7 +285,10 @@ xceadmin config push cluster.bin --controller localhost:9991
 Be aware of these before treating anything above as a production runbook — they are honest gaps in this repo today, not documented-but-hidden features:

 1. **No process supervision is shipped.** No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons. A systemd-unit *sketch* exists in `src/docs/config-replication-design.md` but has a wrong binary name (`xcecontroller` vs the real `xce_controller`) and specifies `Type=notify`, which the controller does not support (it never calls `sd_notify`) — using it as written will hang at startup and time out. If you adapt it, switch to `Type=simple` and fix the binary name.
-2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**: four accounts total — `admin`/`admin` and `op7x2k9`/`CHANGE_ME_dev_only` (both role `cluster_admin`), plus `user1`/`user1` and `user2`/`user2` (role `basic_user`). Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead.
+2. **`src/frontend/apiserver/config/auth.yaml` ships known dev credentials checked into git** (stored as bcrypt hashes since bug-1947, but the passwords are public): four accounts total — `admin`/`admin` and `op7x2k9`/`CHANGE_ME_dev_only` (both role `cluster_admin`), plus `user1`/`user1` and `user2`/`user2` (role `basic_user`). The apiserver logs a warning at startup for each default credential still active and for each plaintext `password:` entry. Don't ship the checked-in file as-is; set real credentials (`go run ./tools/encrypt -action=hash` in `src/frontend/apiserver` prints a bcrypt hash) or an external auth provider instead.
+   - **JWT secret**: set `XCE_JWT_SECRET` (or `authentication.token.secretKey` in `app.yaml`, which ships empty). Without one, the apiserver signs tokens with a random per-process secret and logs a warning: tokens do not survive a restart, and several apiservers behind one endpoint (the nginx sample's upstream) must share the same `XCE_JWT_SECRET`. The old public default `xce-api-key` is treated as unset (bug-1945).
+   - **Logout** revokes the token in memory until its expiry (bug-1946); the revocation is per apiserver process, not persisted and not shared between apiservers.
+   - **Reverse proxy**: `X-Forwarded-For`/`X-Real-IP` are honoured only from addresses in `app.yaml` `security.trusted_proxies` (default none, bug-1948). Behind nginx/HAProxy, list the proxy's address there, or every client is seen with the proxy's IP (IP whitelist, JWT IP binding).
 3. **All controller-bound traffic is encrypted but not authenticated.** Controller↔controller HA replication and every `libxce` client (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller) run over TLS, but `SSL_VERIFY_NONE` is used throughout — no client certificates, no hostname/identity verification. This buys confidentiality and tamper-resistance in transit, not authentication of the peer; anyone who can reach the port and complete a TLS handshake is treated as trusted the moment the transport-level join succeeds.
 4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated. (This is apiserver's REST-facing HTTPS, separate from its TLS-encrypted `libxce` connection to the controller in item 3.)
 5. **`xceadmin config push` against a real cluster** — the decode-crash bug is fixed (2026-07-30, `decode_reply()` unwraps the `Request` envelope), but `--dry-run` still has no real controller-side handler (see Known Limitations item 2).
--- a/src/docs/xce.md
+++ b/src/docs/xce.md
@ -54,6 +54,108 @@ cluster. Common operations:
 Built as a standalone Go binary; communicates with the API server over
 its REST interface.

+## Killing and stopping jobs
+
+Job control follows LSF's TERMINATE and SUSPEND actions:
+
+- **Kill** (`xcecli jobs kill <id>`, no signal given): SIGINT, then SIGTERM,
+  then SIGKILL, one interval apart, each only while the job is still
+  running, so the job can catch SIGINT/SIGTERM and clean up.
+- **Stop** (`xcecli jobs stop <id>`): SIGTSTP first, so the job can catch
+  it, prepare and stop itself; SIGSTOP follows after the interval if it has
+  not stopped. **Resume** sends SIGCONT and cancels a stop still in progress.
+- **Explicit signal**: that signal only, once (`xcecli jobs kill -s
+  <name|number> <id>`, e.g. `-s TERM`, `-s SIGUSR1`, `-s 9`; REST API:
+  `signal` in the control request, kill action only).
+- **Force kill**: SIGKILL at once (`xcecli jobs kill --force <id>`; REST
+  API: `force_kill: true`).
+
+`--signal` and `--force` work with job IDs and with the filters (`-J`, `-q`,
+`-u`, `-a`), and cannot be used together. `stop` and `resume` take no
+signal.
+
+The interval is the API server's `job.terminate_interval` setting
+(`config/app.yaml`, seconds; default 10 when unset or 0, as LSF's
+`JOB_TERMINATE_INTERVAL`; negative refuses startup).
+
+## Job arrays
+
+A job array is a set of identical jobs (elements) submitted at once. XCE
+supports a subset of the LSF array syntax.
+
+### Supported
+
+- **Submitting.** `xcecli jobs submit -J 'name[1-N]' <command>` submits an
+  array of N elements, numbered 1 through N. `-J 'name[1]'` submits a
+  one-element array. Element numbers always start at 1.
+- **Array size limit.** N may be at most the API server's
+  `max_array_size` setting (`config/app.yaml`; default 1000 when unset or 0,
+  never above 32767 — larger values are clamped to 32767 with a startup
+  warning). A larger N is rejected with a message stating the effective limit.
+- **Array name.** The part before `[` is 1 to 4094 characters, each an ASCII
+  letter, digit, `.`, `_` or `-`. No whitespace is allowed anywhere in the
+  expression.
+- **Submit result.** Submission reports the array identity, without an
+  element number, and the number of elements created, e.g.
+  `Job <array-id> is submitted to queue <normal> (N elements).` The REST
+  response carries the same identity and an `array_size` field.
+- **Referring to elements.** Each element is shown and addressed as
+  `<array-id>[k]`, in listings, job detail output, control command results and
+  in log entries that print a job identity in its written form (some log
+  entries still print the raw job number). Element numbers in a selection are
+  at most 32767.
+  - Listing (`xcecli jobs list`) and control commands (`kill`, `stop`,
+    `resume`, `checkpoint`) accept one element, `<array-id>[k]`.
+  - Control commands (`kill`, `stop`, `resume`, ...) also accept a range,
+    `<array-id>[a-b]` (`a` equal to `b` allowed). Queries accept a range only
+    by job name, e.g. `-J 'name[a-b]'`.
+  - `-J 'name[k]'` selects element k of every array named `name`.
+  - An identity without `[...]` selects every element of that array (and any
+    ordinary job with that identity), as before.
+  - In listing and control commands, a bare element job number (for example
+    the value of `XCE_JOBID`) is rejected: name the element as
+    `<array-id>[k]`.
+  - `xcecli jobs peek` and `xcecli jobs restart` take one job: a job ID,
+    or one element as `<array-id>[k]` (not a range). `peek` also accepts
+    an element's own job number (`XCE_JOBID`). Restarting a bare array
+    identity is refused: restart one element at a time.
+- **Element number in the job's environment.** A running element finds its
+  own number in `XCE_JOBINDEX` (1..N). An ordinary job gets
+  `XCE_JOBINDEX=0`. A user-supplied `XCE_JOBINDEX` is overridden, with a
+  warning in the worker log. `XCE_JOBID` is the element's own job number, as
+  before.
+- **Test harness.** `xce_test jobsub -a N` submits elements 1 through N
+  (2 <= N <= 32767, regardless of `max_array_size`; `-a 1` submits an
+  ordinary job). `$jobid` is then the array identity and `$jobid[k]` names
+  element k (for example `expect-job $jobid[3]`).
+
+### LSF array features not supported
+
+Each of these is rejected at submission with a message naming the problem; no
+job or element is created.
+
+- **Starting numbers other than 1**, e.g. `name[0-9]` ("element numbers start
+  at 1") or `name[1001-2000]` / `name[5]` ("only name[1-N] and name[1] are
+  currently supported").
+- **Index lists**, e.g. `name[1-5,10]` or `name[1,3,5]` ("only a single range
+  name[1-N] is supported").
+- **Index steps**, e.g. `name[1-10:2]` (same message).
+- **`%N` running limits**, e.g. `name[1-100]%5` ("running limits are not
+  supported").
+- **Interactive arrays**: an interactive job (`-I`) whose name contains `[`
+  or `]` is rejected ("interactive jobs cannot be arrays").
+
+These LSF features are not available either:
+
+- **Per-element `%I`/`%J` file-name substitution.** `%I` and `%J` in file
+  names (`-i`, `-o`, `-e`, output directory, working directory) are replaced
+  once, at submission, with the first element's values: `%I` becomes `1` and
+  `%J` becomes `<array-id>[1]` for every element. All elements therefore write
+  to the same files unless the job script chooses its own file names from
+  `XCE_JOBINDEX`.
+- **`LSB_JOBINDEX` / `LSB_JOBID` variables** are not set. Scripts read
+  `XCE_JOBINDEX` (element number) and `XCE_JOBID` (job number) instead.
+
 ## shell

 An interactive bash shell used for troubleshooting and inspection. Not