18 sub-tasks in 6 groups (libxce primitives + regression script
skeleton; controller; scheduler/worker/harness; xceadmin; integration;
docs). Each C task adds its failing regression-script case before the
change. Requirements, design and tasks approved.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Iterated design review until clean (GO):
- Controller main() calls mylog_fini() before every early return, so
log lines written by tlog's background thread survive the exit.
- cluster_init_phase1() keeps its candidates_count > 0 guard around the
new xce_peers_find_identity() call (standalone stays my_priority 0).
- xceadmin adds the "(from $VAR)" source note only to network errors,
not to local cluster-bin file errors (OSError with filename set).
- Regression script: scratch --workdir/--certdir/--logfile per daemon,
refuses to run if a local cluster is up (port 9990, worker socket),
--sim case asserts the sim branch ran, 2.3 patches follow_redirects,
2.8 malformed-entry cases use a readable cluster-bin file, wrapper
test expects PEERS_STR form.
- Two existing scripts that start xce_scheduler without --peers will
unset XCE_CONTROLLER_PEERS; --interactive wording corrected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Design review (NO-GO -> GO) and iterated requirements review until clean:
- Controller refuses to start (non-zero exit, stderr, before any network
I/O) when an env-supplied XCE_CONTROLLER_PEERS lacks its own name (1.6);
new libxce xce_peers_find_identity() replaces both identity loops in
controller/cluster.c so the check and my_priority/listen port agree.
- deploy-ha.sh's generated scripts/xceadmin wrapper exports
XCE_CONTROLLER_PEERS so an ambient value cannot redirect it (Req 4).
- Opt-out is unsetting/emptying the variable (1.7); "No --peers" log
lines say it was unset or empty (1.8); docs cover sudo/systemd not
inheriting it (3.1).
- Requirement 1 defines explicit candidate list / resolution point /
own name so harness (set masters) is covered precisely; error messages
name the value's source incl. validation failures (1.4).
- xceadmin: name= stripping and a clean malformed-entry error apply to
every --controller source; errors name the env var source (2.2, 2.8).
- Test plan fixed against real code (stderr vs stdout, log texts,
xcecd_virtual, --norc, free ports, failover order test).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- apiserver peek (parseJobIDParam): <array-id>[k] now resolves to the
element; it used to drop [k] and look up the array identity itself
(and panicked in parseArrayName on some inputs). A job number still
names that job exactly (an element by its own XCE_JOBID); ranges and
invalid selections are rejected.
- apiserver restart: the decimal-only pre-check (isValidJobId, now
unused and removed) rejected the written form c-a-ts-seq[k];
ParseIdentitySelection, already used for the lookup, validates it.
- xcecli jobs restart converts its argument with ToAPIJobID like the
control commands, so written IDs work and invalid ones fail locally.
- docs: the peek/restart caveat in the Job arrays section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It logged in to a hard-coded http://localhost:8080 with fixed
credentials, so 'go test ./...' failed with connection refused. It now
skips unless XCE_LOAD_TEST_URL is set, with the credentials in
XCE_LOAD_TEST_USER/XCE_LOAD_TEST_PASSWORD; 'go test ./...' is green.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
REGEX_JOBID, REGEX_JOBARRAY_ID and N_MAX_FIELDS_JOBID served the old
regex-based parse_jobid(), replaced by parse_jobsel() in
job-array-one-based-index task 1.2; nothing references them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Formatting only (gofmt -w on the 29 non-vendored, non-generated files
gofmt -l reported); swag-generated apiserver/docs/docs.go left as is.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The duplicate-id numbering lives in OpenWolf's post-write hook, which
openwolf update overwrites; auto_detect=false survives updates and the
pre-commit stale-index check catches any stray hook write.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pc13v capture's snapshot holds format_resreq() dumps instead of
resreq text (written before 077b1213), so the simulator cannot rebuild
that array job's request. Re-verified with fresh captures from the
current scheduler, jobs in the snapshot: 402 decisions, 0 MISMATCH.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Re-verified 2026-10-01: test_jobj_hardening and the full xcelib_test run
clean with detect_leaks=1; meson's libxce test (abort_on_error=1) OK.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
clitest_wait_for_exit/clitest_wait_for_text added CLITEST_POLL_SLICE_MS
per loop instead of measuring time. Once xce_test closes its pty after
'quit', poll() returns at once on every call, so the 5 s budget was used
up within milliseconds while the (ASAN, loaded) child was not yet
reapable -- measured: real exit within 6 ms while the counter reached
58800 ms. The waits now use a CLOCK_MONOTONIC deadline, and
clitest_pump_once naps 10 ms when poll wakes with nothing to read.
Under 16 parallel runs: 12/48 runs failed before, 0/96 after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_job_cpu_stats() (misc.c), run after each usage report:
- each host's cpu_rate = CPU-seconds gained / wall seconds since its
previous report; the job's cores = the sum over hosts
- peak_cpu = highest job cores seen (peak_cpu_time = it rounded, the
wire field is an integer); peak_cpu_duration = the interval that set it
- avg_cpu_efficiency = cpu_time / elapsed / slots, peak_cpu_efficiency =
peak_cpu / slots, mem_efficiency = highest used_mem / rusage[mem]
request (job-wide total, per task by default), all in percent
aggregate_job_runusage() no longer treats cumulative cpu_time as a peak.
New fields host sample_time/cpu_rate and job peak_cpu are appended to
the capnp structs (old WAL entries still decode) and copied by
merge_job_runusage(), so a promoted standby keeps the rate baselines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
RestartJobByID looked the job up with a job_id CompCondition, which the
store never evaluates, and restarted jqueryReply[0] -- the first stored
job -- for any ID, including unknown ones. It now parses the ID with
ParseIdentitySelection, fetches the rows by key and matches them with
foundJob, as ControlJobByID does: an ordinary job or <array-id>[k] is
restarted, an unknown job or element is not found, and a bare array
identity (several elements) is refused. Restart goes through the
generateJobID/submitJob seams so the handler fixture can test it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follows page_token with limit=3 over the handler fixture. Before
efff6b65 the token's float-rounded array-element ID matched no row and
every page restarted at row 0 (the original repeat-the-first-page
report); it now pages through every row exactly once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller forwards a kill as soon as it has dispatched a job, but
the worker queues the start and only registers the job (and, for the
real worker, connects its IPC client) once it starts -- seconds later
for the first jobs after a worker starts. A signal arriving in between
was dropped ("Failed to find worker"), although the API reported it
accepted, and the job ran to completion.
New pending_signal table: the start handler records the job, a signal
for a queued or still-starting job is kept there, and it is delivered
when the job reaches RUNNING (virtual: killed_signal; real: IPC). A
failed start or a job that ends drops the entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The virtual worker's job-end decision lived inside the branches of the
systemd exit-code check, and only the exit-code-0 branch looked at
killed_signal. Without a system bus that call returns -1, so a kill was
recorded but ignored until the job's sleep time elapsed. A virtual job
has no systemd unit: decide its end from the kill and the timer alone,
before asking systemd.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller acknowledges job dispatch and pend-reason requests with
status 0 before looking the job up, and the scheduler relies on that
early acknowledgement, so the harness now looks the job up first
(job_exists_or_report) and reports "not found" like the other
single-job commands (job-array-one-based-index Req 4.7, 5.13).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
findPageTokenIndex decoded the token's unique_job_id into a float64;
array element IDs set bit 62, where float64 spacing is 1024, so paging
skipped rows or restarted at row 0 -- an endless loop for callers that
page until the token is empty (control/query of arrays over 1000
elements). Decode with UseNumber and keep integers as int64.
Found by the job-array-one-based-index feature validation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
