cJobToModel pre-filled main_reason_name with pendReasonName(main_reason),
so a detail-only record (main_reason 0, the allocator's ordinary shortfall
shape) arrived at rawJobToModel named "unknown(0)" — and the promotion that
derives the overall reason from the first detail row requires an EMPTY
name. The promotion's own tests passed because they build rawJobData
directly; the live C path never hit it. Found by the advisor P0
verification: jobs whose res_reasons named no_resource read
main_reason_name=unknown(0) over the API.
main_reason 0 now leaves the name empty for the promotion to fill; a
non-zero main reason keeps its mapped name unchanged.
Co-Authored-By: Claude Code <noreply@anthropic.com>
handleNotifyData had no case for NotifyData_data_jobPidInfo, so the
controller's start-report push (apiserver_job_start_fun, fired from
HOOK_DETAIL_JOB_START the moment WAL_JOB_START commits) was silently
dropped. interactive_job_url then stayed invisible until the next
periodic rusage-driven job UPDATE re-delivered the full row — up to
10s later (job_rusage_collect_interval), the dominant chunk of an
-I --tty job's submit-to-first-output latency (measured ~6s on t1,
server-side work itself is ~0.6s).
Add OnJobPidInfo to the notification interface and DefaultHandler:
it merges the worker-reported pids/pgids and the exdata {"url": ...}
onto the already-stored job record (get-mutate-put, mirroring
OnJobFinished) and publishes a job.updated event. The CGO decode
side is factored into cPidInfoToRaw/pidInfoJSONB, shared with the
existing xce_job_t conversion.
Cluster-verified on t1: URL is pollable ~0.6s after submit
(previously ~11s), full submit/run/finish lifecycle clean, all
daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The worker packs the interactive session URL (ttyd/XJOB) into the start
report's exdata as {"url": ...}; the controller stores it on
job->job.pidinfo->exdata and the capnp Job wire carries pidinfo (pidinfo@9,
JobPIDInfo.exdata@3) — but apiserver's cJobToModel never mapped pidinfo, so
models.JobInfo.InteractiveJobURL (declared only in the query-field whitelist
JobAdditionalFields) had no producer and interactive_job_url was permanently
empty in API responses. xcecli's waitForInteractiveURL/openBrowser flow could
never fire.
- models.JobInfo: add InteractiveJobURL (json interactive_job_url, matching
the JobAdditionalFields whitelist name).
- convert_pure.go: rawPidInfo (pids/pgids/exdata) + interactiveURLFromExdata;
rawJobToModel marshals pids/pgids into PIDInfo (pid_info is a JSONBJFields
entry, parsed by processJobData) and promotes the exdata URL to
InteractiveJobURL.
- convert.go: cJobToModel extracts job.pidinfo via pidinfo_pid/pidinfo_pgid
cgo helpers (CGO cannot index the C arrays directly).
Cluster-verified end to end on t1 (fresh apiserver deployed): tty job
(xcecli -I --tty equivalent, exdata {"interactive":1}) now returns
interactive_job_url=http://<worker>:9000 and a populated pid_info; non-tty
jobs return ""/no pid_info. All daemons healthy, zero errors in window.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).
Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.
Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
queue refuses new submissions only; pending jobs still run) --
supersedes our QUEUE_CLOSED preason on the dispatch path;
sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
rounds (sched_stash_config/sched_apply_pending_config), applied at
the same main-thread round boundary that preserves bug-1513
serialization
- bug-1760: sched_check_pending_rejoin exposed +
xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
(superset of our bug-1500 execuser fallback; our job-typed
resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
src/harness/test.c; controller suite split into
controller_test.c + fixtures; their clitest fixtures spawn xce_test
mid-suite under `meson test -j`
Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
gate re-ported into their relocated allocator (sched_election.c)
after plugin_job_election, with attach on the allocator thread
only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
merge initially re-carried our first-detail-add-promotes-main_reason
fallback in add_job_preason (4e2ac86), but their whole trace/verify
stack treats "detail-only => main_reason 0" as a contract --
sched_trace_capture counts main_reason as its own occurrence
("Expected 2 Was 3" in test_trace_capture), the outcome renderer
prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
and the committed .replay/.sim_verify baselines encode main_reason=0
(test_repl_trace_baseline byte compare, test_trace_equivalence
off-vs-traced). The fallback is dropped from the shared lib;
the user-visible "overall reason" survives as an API-layer view:
rawJobToModel (convert_pure.go) promotes the first detail row to
main_reason_name when main_reason is 0 (main_reason VALUE marshals
unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
(their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
resend, xcecli_force_reconnect + their is_connection_broken both
exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
/xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)
bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
(testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
harness_test.c -- hundreds of cases per process) call
case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
dirs are swept immediately in tearDown (after engine reaping), FAILED
cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
disables. One full controller_unit_test run was observed to leak 12G+
to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
failures
- harness_unit_test_deps links the shim library as well
Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
detail-keeps-main-0 contract; new tests cover limit-code detail
rows and explicit-main-wins-over-later-details; apiserver
convert_test.go gains the detail-only name-promotion cases
Co-Authored-By: Claude Code <noreply@anthropic.com>
Cause: modelJobToSubmit() filled only cmdline, execuser, launch_type and
pre/post-exec of job_startinfo_t, so the extended data the submit handler
builds (env, *_limit incl. -W run limit, file transfers, interactive mode),
the cwd and the exec home never reached the controller or the worker.
Fix: map the start info in a pure submitStartInfo() (exdata from
ExtendData, execcwd from ExecCWD or CWD, exechome from ExecHome) that
modelJobToSubmit copies into the C struct; decode exdata back in
cJobToModel/cJobArrayToModels so the store keeps ExtendData and CWD.
Reject extend_data the worker's parse_exdata() cannot take (not a JSON
object, over EXDATA_LENGTH-1 bytes) and drop an empty transfer list
instead of sending a null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1934: every job-permission check read a single scope and compared it
only with "self", so a job:view or job:control grant scoped to self_ug acted
like "all" (list, get by ID, control), and self + self_ug collapsed to self.
userJobScope() now collects all of a user's grants of a job permission
before deduplication and adds them up (all; self; self_ug = the user and
everyone sharing a user group). The job list appends a submit_user filter
for narrower scopes, get-by-ID keeps covered rows in a new slice (removing
from the ranged slice skipped rows) and answers 403, both control paths
and job output check the owner against the scope. User groups now reach
the apiserver: the config push converts cluster_config_t.ugroups and
OnUserGroupsUpdated() replaces the new store (nested groups, all,
exclusions). The default basic_user now also sees its user-group mates'
jobs, as its configuration says.
bug-1935: project_name and license_project are accepted job filters, so
xcecli jobs list -P works.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller computed run usage (CPU time, memory, CPU peak and
efficiency, LSF bjobs -l definitions) but the capnp Job struct had no
field for it, so the job pushed to the apiserver never carried it, and
snapshots (the same struct) dropped it: a job restored from a checkpoint
lost its accumulated usage.
- controller.capnp: Job.runUsage @13 (JobRunusage); old entries decode
with no run usage.
- apiserver: map run_usage into JobInfo; max_mem, avg_mem, used_mem,
cpu_time, peak_cpu, peak_cpu_time, peak_cpu_duration, mem_efficiency,
avg_cpu_efficiency, peak_cpu_efficiency become queryable columns.
- xcecli: -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK,
CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now
renders too), and a MEMORY USAGE / CPU USAGE block in jobs list -l.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
modelJobToSubmit() never set job_startinfo_t.execuser when converting
a submitted job to the C struct sent to the controller/worker, so
every job ran as the worker's 'nobody' default and failed to exec
(exit 126). Default execuser to the submitting user via a new
resolveExecUser() pure helper, and free the newly allocated C string.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
NJOBS/RUN/SSUSP/USUSP (xcecli hosts list -l) and EXEC_HOST (xcecli jobs
list) always showed a hardcoded 1 slot per host regardless of a job's
actual per-host allocation -- rawJobToModel()'s ExecHosts JSONB builder
never read the real per-host "slots" resource from the C job allocation,
so the already-correct downstream slot-counting logic (computeHostSlotUsage,
bug-1743) and EXEC_HOST's "N*host" formatter never had real data to work
with. convert.go now extracts each host's real slots allocation via a new
CGO indexing helper, mirroring the scheduler's own get_nslots_in_alloc()
convention.
Also wires hosts list -l's "CURRENT LOAD USED FOR SCHEDULING" table (tmp/
swap/mem/slots and friends) to the host's real live resource values --
previously entirely hardcoded placeholder strings, never connected to any
backend data.
modelJobToSubmit/freeJobSubmit (submit direction) and
cJobToModel/cJobArrayToModels (receive direction, both single-job and
array-job paths) now carry from_host across the CGO boundary.
rawSubmitData/rawJobToModel (the pure-Go, CGO-free mapping layer) gain
the field too, completing the round-trip so JobInfo.FromHost reflects
whatever the controller actually has -- which, as of the prior two
commits, is now always the client-asserted hostname. backend/handler.go
needs no change: its existing blind overwrite in OnJobUpdated is
correct by construction once every controller push genuinely carries
the right value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
P0 (done, cluster-verified):
- main_reason was always 0: add_job_preason only set it when BOTH host and res
were NULL, but the allocator always attaches host/res detail rows. Now the
first detail add wins as main_reason, so the API's main_reason_name is
meaningful (was "unknown(0)"). Verified live: PEND jobs now report
main_reason=3/match_fail + per-host reason_name.
- limit plugin attach: host-based quota hit (hg1_slots_per_host) -> LIMIT_HOST
+ limit name; global quota hit (ug1_slots, ...) -> LIMIT_GLOBAL + limit name.
Previously both blocks were silent (job just stayed PEND).
- Extended the PREASON_* enum (was 3 codes) with LSF-style classes:
LIMIT_GLOBAL/LIMIT_HOST/QUEUE_CLOSED/QUEUE_DRAINING/TIME_WINDOW/HOST_UNAVAIL.
apiserver pendReasonName maps them; host/res detail rows now carry reason_name.
P1 (implemented):
- queue status: sched_queue_is_ready no longer a stub — closed/inactivated
queues are not electable; the fcfs election attaches QUEUE_CLOSED /
QUEUE_DRAINING to every PEND job in a skipped queue.
- worker filter: sched_try_alloc_on_host_ skips UNAVAIL/CLOSED workers and
attaches HOST_UNAVAIL instead of trying to allocate on them.
Unit tests: preason suite updated for the main_reason change + 3 new tests
(limit host/global, first-add-wins); 295 lib tests pass under ASAN.
apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
The pending_reason field was declared on models.JobInfo but never populated:
scheduler computed PEND reasons and the controller stored them on
job->pend_reasons, yet the data stopped there. Two gaps closed:
1. controller/apiserver.c: apiserver_job_hook_fun now handles
HOOK_DETAIL_SETPREASON by pushing the updated job to API servers (reusing
apiserver_job_update_fun). Without this, a job whose PEND reasons change
never reached the API server — only JOB_NEW/UPDATE/START/EXIT events did,
and setpreason fired none of those.
2. backend/convert.go + convert_pure.go: cJobToModel / rawJobToModel now
extract xce_job_t.pend_reasons (main_reason + host/res reasons) and
marshal them into models.JobInfo.PendingReason as JSONB. models/jobs.go:
pending_reason added to JOB_DEFAULT_COLUMN so default job queries return
it (it was only in JSONBJFields, never selected).
Verified live on t1/t2/t3: a job with resreq "type == nosuchhost" stays
PEND and returns
"pending_reason":{"main_reason":0,"host_reasons":[{"host":"t1","reason":3},
{"host":"t2","reason":3},{"host":"t3","reason":3}]}
(main_reason=3 PREASON_MATCH_FAIL per host) via both /jobs list, /jobs/:id
and the SSE stream. 4 new unit tests (rawJobToModel pendReason marshal +
name mapping), all apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
Two stacked defects in the apiserver job-control link, both of which kept
hold/release/cancel from ever reaching the controller:
1. controllers/jobs.go: validateJobControl and foundJob asserted
jobinfo.Data["array_idx"].(int64), but StoreToOrderMaps produces
array_idx as int (models.JobInfo.ArrayIdx is declared int) -> panic
-> Gin Recovery -> HTTP 500 before backend.ControlJob ran. Add
normalizeJobInt64 / jobArrayIdx helpers and use them in both
functions (job_status already had this fix via normalizeJobStatus;
array_idx was missed).
2. backend/clib.go: backend.ControlJob set xce_jobctrl_t.action to the
models.JOB_* constant (JOB_STOP=1 / JOB_RESUME=2 / JOB_KILL=3 /
JOB_CHKPNT=8), but that field is the capnp JobControl union
discriminant (JobControl_u_siginfo=0 / JobControl_u_ckptoption=1).
The controller's job_ctrl() selected the wrong union branch (or
none, since encode_JobControl drops the union pointer for
non-0/1 discriminants) and rejected every control with -EINVAL.
Add jobCtrlDiscriminant() and use it.
Also: resolveExecUser was defined in the cgo-only convert.go but is
exercised by a non-cgo test, so `make test` (CGO_ENABLED=0) failed to
build the backend package. Move it to convert_pure.go so it compiles in
both modes.
Tests: 3 controller regression tests (validateJobControl allowed /
not-allowed branches, foundJob array-child branch) + 1 backend
jobCtrlDiscriminant test. make test fully green. Cluster E2E verified:
hold -> PSUSP, release -> PEND, cancel -> EXIT on a PEND job, no 500,
no -EINVAL, all daemons healthy.
Co-Authored-By: Claude <noreply@anthropic.com>
modelJobToSubmit built the C job_startinfo_t without setting execuser, so the
controller forwarded a NULL execuser in the startjob request and the worker
ran every job as "nobody". Now the submit carries the resolved exec user: an
explicit exec_user wins, otherwise the job runs as the submitting user.
- add resolveExecUser() (plain-Go, unit-testable) + TestResolveExecUser
- set si.execuser in modelJobToSubmit, free it in freeJobSubmit
Note: deploying this exposed an ASAN heap-use-after-free in the apiserver's
submit/notification path (strncpy reading freed memory); the fix is committed
but needs the memory-lifecycle investigation before it can ship safely.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the Go/CGO receive path for Data.jobArray, mirroring task 5.3's
scheduler-side C implementation: rawJobArrayToModels() (pure-Go
base_jobid + i expansion), cJobArrayToModels() (CGO adapter sibling to
cJobToModel), dispatchJobArray() (dedup via the same store.Jobs.Has()
the ordinary Data_data_jobs path already writes through, single
confirmation log only when at least one element was newly
dispatched), and a new case C.Data_data_jobArray in handleDataset.
Elements route through OnJobUpdated, matching the ordinary single-job
push's own routing. The CGO boundary needed no changes -- both
C.xce_job_submit_t and C.Data_data_jobArray were already exposed from
prior tasks, including task 5.3b's extract_data() fix which this
receive path depends on identically to the scheduler's.
store.ArrayJobs (the array-level rollup aggregate) is deliberately
left unpopulated -- confirmed out of this spec's scope, deferred to a
future spec per requirements.md's Boundary Context.
Task 5.4 of job-array-single-wal-record (Requirements 6.2, 6.4, 6.5, 8.4).
Add OPCode_notify handler in go_push_handler top-level switch
(not inside handlePush) to receive resource value updates from
the controller via xce_notify_data_t/schedUpdateResvals.
- Add 8 CGO union accessors for xce_val_t (which, i32, i64,
u32, u64, fval, str, bool) since CGO cannot access union fields
- Add handleNotifyData() dispatching NotifyData_data_schedUpdateResvals
events, iterating xce_resval_list_t and calling cResvalToDistribution
- Add cResvalToDistribution in convert.go with 7 separate switch
cases for Value_u_* union discriminants (no combining i32/i64)
- Add rawResvalData + rawResvalToDistribution pure-Go version
in convert_pure.go for testability without CGO
- Add 10 TDD tests covering all 7 value types, nil val guard,
unknown discriminant, and boolean false
- All 42 backend tests passing
Wires the remaining cluster_config_t data into the notification pipeline:
- dispatchConfigPush() extracted as a pure-Go helper (CGO-free, testable)
that drives OnQueueUpdated, OnResourceUpdated, OnHostConfigUpdated, and
OnClusterUpdated from a single config push event
- handleConfigPush() delegates to dispatchConfigPush via the new helper,
eliminating the previous inline loops
- OnHostConfigUpdated and OnClusterUpdated added to NotificationHandler
interface and implemented with partial-merge semantics in DefaultHandler
- rawHostConfigData/rawHostConfigToModel and rawClusterData/rawClusterToModel
added to convert_pure.go with their CGO extraction counterparts in convert.go
Safety fixes from code review:
- OnHostUpdated, OnHostConfigUpdated, OnClusterUpdated now copy *incoming to
a local merged value before mutation, closing the TOCTOU window between
store.Get and store.Put under concurrent pushes
- cClusterConfQueues and cClusterConfResources gain nil-pointer guards
matching the existing guard in cClusterConfHostConfigs
- handleConfigPush gains an explicit gHandler nil check
- cClusterConfToClusterInfo returns nil on empty clustername (prevents phantom
empty-key write to store.Clusters); handleConfigPush logs a warning when nil
- 36 unit tests passing (CGO_ENABLED=0)
