Render -X jobs' windows on the user's own X server with zero client
install, mirroring the ssh -X experience: the job command starts with
DISPLAY pointed at a per-job TCP listener on the worker, and the user's
xcecli relays that listener's byte stream to their local X server.
- xcecli: new -X/--x11 flag (interactive=2 + exdata "x11":1). After
submit it fetches the per-job MIT-MAGIC-COOKIE-1, registers it in the
user's Xauthority, then bridges one xattach websocket to the local X
server address parsed from $DISPLAY (unix socket or TCP). The cookie
entry is removed when the session ends. Pre-flight checks report
missing DISPLAY/xauth before submit.
- apiserver: GET /xce/v1/jobs/:id/xattach relays websocket binary frames
to the worker's forwarding port (raw TCP, no ttyd). Close codes match
the ttyd attach (4000 unreachable, 4001 ended). The job-finished event
ends the relay. Teardown closes both hops as soon as either pump
exits, so a client disconnect propagates instead of deadlocking in a
blocking read.
- worker: XJOB jobs with exdata x11=1 skip VNC, listen on 6100-6199,
generate a per-job xauth file (/tmp/job_<id>_xauth, worker-side since
the worker has no session), export DISPLAY/XAUTHORITY to the command,
and bridge the app connection (loopback) with the relay connection
(non-loopback; roles classified by source address, not arrival order).
A 60 s no-client watchdog reaps jobs nothing ever attached to; the
v1 single-connection boundary refuses a second X connection loudly.
t1-verified per the plan matrix: window rendering (xclock/xeyes on
Xvfb), normal end -> DONE + xauth removed, no-client 60 s reap (137),
kill -> immediate teardown with no residue, -I/-I --browse regression,
daemon health, missing-DISPLAY error, second-connection refusal.
(--xjob VNC on t1 fails for lack of a vncserver binary; pre-existing.)
Co-Authored-By: Claude Code <noreply@anthropic.com>
An attach job (xcecli -I without --browse) that no client ever connects
to sat in RUN forever: ttyd only starts the job command on the first
websocket client, so with no client the command never runs, ttyd never
exits, and the job holds its ttyd, port and per-job worker indefinitely.
New exdata key "attach":1, set by the CLI for TTY jobs that are not
--browse and carried through the apiserver's extendDataMap to the
worker's transfer_info->attach_mode. For such jobs the worker starts a
1 s watchdog when the job process starts:
- no client attached within 60 s -> reap ttyd (SIGKILL to the monitored
tree), job EXITs with status message "No client attached within 60 s"
- client came and the command ran and exited -> give ttyd 5 s to stop
on its own (-Q), then reap: "Command exited; ttyd did not stop on its
own"
The reap lands in the regular process_exit_callback path (JOB_EXITED,
status reason, post-exec, worker exit), so kill and cleanup behave like
any other job termination.
Field-critical detail: uv_proc_get_pid returns the "/bin/sh -c ttyd ..."
wrapper pid, and sh does not reliably exec -- on the field ttyd runs as
a child of the wrapper. Judging the wrapper's children would see ttyd
itself as "the command has started" and never reap. The watchdog
resolves the real ttyd pid first (/proc/<pid>/comm walk down through
the sh wrapper, capped depth) and judges client activity by the real
ttyd's children; killing still goes through the monitored wrapper tree
so both topologies (exec'd or forked) are covered.
Web-viewed sessions (--browse / DISPLAY) are untouched: no attach
exdata, no watchdog -- ttyd stays until a browser connects or the job
is killed, as before.
Attach failure on the CLI side prints that the job will be cleaned up
by the worker if no client attaches within 60 s.
Tests: xcecli (attach flag matrix), apiserver controllers (attach
exdata end-to-end in ExtendData), worker native suites; live t1
cluster: normal attach completes DONE with no residue, unattached job
EXITs at ~60 s with the reason in the worker log and port released,
Ctrl-C-then-wait completes when the command ends, --browse job stays
RUN past 60 s and kills cleanly.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The worker packs the interactive session URL (ttyd/XJOB) into the start
report's exdata as {"url": ...}; the controller stores it on
job->job.pidinfo->exdata and the capnp Job wire carries pidinfo (pidinfo@9,
JobPIDInfo.exdata@3) — but apiserver's cJobToModel never mapped pidinfo, so
models.JobInfo.InteractiveJobURL (declared only in the query-field whitelist
JobAdditionalFields) had no producer and interactive_job_url was permanently
empty in API responses. xcecli's waitForInteractiveURL/openBrowser flow could
never fire.
- models.JobInfo: add InteractiveJobURL (json interactive_job_url, matching
the JobAdditionalFields whitelist name).
- convert_pure.go: rawPidInfo (pids/pgids/exdata) + interactiveURLFromExdata;
rawJobToModel marshals pids/pgids into PIDInfo (pid_info is a JSONBJFields
entry, parsed by processJobData) and promotes the exdata URL to
InteractiveJobURL.
- convert.go: cJobToModel extracts job.pidinfo via pidinfo_pid/pidinfo_pgid
cgo helpers (CGO cannot index the C arrays directly).
Cluster-verified end to end on t1 (fresh apiserver deployed): tty job
(xcecli -I --tty equivalent, exdata {"interactive":1}) now returns
interactive_job_url=http://<worker>:9000 and a populated pid_info; non-tty
jobs return ""/no pid_info. All daemons healthy, zero errors in window.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).
Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.
Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
queue refuses new submissions only; pending jobs still run) --
supersedes our QUEUE_CLOSED preason on the dispatch path;
sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
rounds (sched_stash_config/sched_apply_pending_config), applied at
the same main-thread round boundary that preserves bug-1513
serialization
- bug-1760: sched_check_pending_rejoin exposed +
xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
(superset of our bug-1500 execuser fallback; our job-typed
resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
src/harness/test.c; controller suite split into
controller_test.c + fixtures; their clitest fixtures spawn xce_test
mid-suite under `meson test -j`
Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
gate re-ported into their relocated allocator (sched_election.c)
after plugin_job_election, with attach on the allocator thread
only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
merge initially re-carried our first-detail-add-promotes-main_reason
fallback in add_job_preason (4e2ac86), but their whole trace/verify
stack treats "detail-only => main_reason 0" as a contract --
sched_trace_capture counts main_reason as its own occurrence
("Expected 2 Was 3" in test_trace_capture), the outcome renderer
prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
and the committed .replay/.sim_verify baselines encode main_reason=0
(test_repl_trace_baseline byte compare, test_trace_equivalence
off-vs-traced). The fallback is dropped from the shared lib;
the user-visible "overall reason" survives as an API-layer view:
rawJobToModel (convert_pure.go) promotes the first detail row to
main_reason_name when main_reason is 0 (main_reason VALUE marshals
unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
(their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
resend, xcecli_force_reconnect + their is_connection_broken both
exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
/xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)
bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
(testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
harness_test.c -- hundreds of cases per process) call
case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
dirs are swept immediately in tearDown (after engine reaping), FAILED
cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
disables. One full controller_unit_test run was observed to leak 12G+
to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
failures
- harness_unit_test_deps links the shim library as well
Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
detail-keeps-main-0 contract; new tests cover limit-code detail
rows and explicit-main-wins-over-later-details; apiserver
convert_test.go gains the detail-only name-promotion cases
Co-Authored-By: Claude Code <noreply@anthropic.com>
Cause: CreateJob accepted in_file, out_file, err_file, out_dir,
asked_hosts, begin_time, term_time and transfer_files and returned
success, but neither the controller's submit data/start info nor the
worker carries any of them (the worker never stages transferred files),
so they were silently dropped. Exec home had no submit field at all, and
a relative cwd would have been resolved against the worker unit's
directory.
Fix: new preCheckSubmitOptions() runs right after binding, before file
uploads and ID allocation, and answers 400 naming every unsupported
option that is set; cwd and the new exec_home field must be absolute and
fit the worker's job_assignment_t buffers (1023 bytes). exec_home goes
to JobInfo.ExecHome and from there (submitStartInfo) to the start info's
exechome. num_files already reaches exdata as nofile_limit; a handler
test now pins it. The array %J/%I submit-time substitution test (9.4)
checks cwd instead of out_file/err_file, which are now refused. Swagger
regenerated with `swag init -g cmd/main.go -o docs`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- src/docs/xce.md: explicit signal and force kill are REST-only (signal,
force_kill); xcecli has no signal option yet.
- src/docs/deployment.md: app.yaml row lists the job block
(max_array_size, terminate_interval) and trusted proxies.
- xcecli jobs kill/stop/resume help describes the signal sequences.
- REST: control endpoint annotations and the signal/force_kill model fields
describe the behaviour (swagger output not regenerated: swag fails on an
unrelated annotation).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A plain kill sent one SIGTERM and never escalated, and a plain stop sent
one uncatchable SIGSTOP. Following LSF's TERMINATE and SUSPEND actions, a
plain kill now sends SIGINT, then SIGTERM, then SIGKILL, and a plain stop
sends SIGTSTP (the job can catch it, prepare and stop itself), then SIGSTOP
if it has not stopped; each next signal follows after the grace period and
only while the job is alive (or not yet stopped). A resume cancels a
pending stop; the job's exit ends any sequence. An explicit signal is
delivered once; a force kill is SIGKILL at once.
- JobSignal gains grace (capnp @2, xce_signal_t.grace): > 0 asks for the
sequence, in seconds between signals.
- apiserver: job.terminate_interval (default 10 s, LSF's
JOB_TERMINATE_INTERVAL); jobCtrlParams sends kill = SIGKILL and stop =
SIGSTOP with that grace when no signal is given; the handler no longer
pre-fills a default signal.
- worker: grace travels through the job-control task, the kept-signal table
(pending_signal_*_ex) and the IPC message; the job process runs the
sequence (job_signal_seq_*, uv timer) and checks stop state in /proc.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
worker_signal_callback() sent SIGTERM for a KILL request without force and
SIGSTOP for any TSTP request. job_signal_to_deliver() (new job_signal.c) now
delivers the requested signal; only a forced terminate escalates to SIGKILL.
The apiserver's default signal for a plain stop becomes SIGSTOP (it was
TSTP and relied on the worker's substitution), so a plain stop still
suspends jobs that ignore SIGTSTP.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1934: every job-permission check read a single scope and compared it
only with "self", so a job:view or job:control grant scoped to self_ug acted
like "all" (list, get by ID, control), and self + self_ug collapsed to self.
userJobScope() now collects all of a user's grants of a job permission
before deduplication and adds them up (all; self; self_ug = the user and
everyone sharing a user group). The job list appends a submit_user filter
for narrower scopes, get-by-ID keeps covered rows in a new slice (removing
from the ranged slice skipped rows) and answers 403, both control paths
and job output check the owner against the scope. User groups now reach
the apiserver: the config push converts cluster_config_t.ugroups and
OnUserGroupsUpdated() replaces the new store (nested groups, all,
exclusions). The default basic_user now also sees its user-group mates'
jobs, as its configuration says.
bug-1935: project_name and license_project are accepted job filters, so
xcecli jobs list -P works.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller computed run usage (CPU time, memory, CPU peak and
efficiency, LSF bjobs -l definitions) but the capnp Job struct had no
field for it, so the job pushed to the apiserver never carried it, and
snapshots (the same struct) dropped it: a job restored from a checkpoint
lost its accumulated usage.
- controller.capnp: Job.runUsage @13 (JobRunusage); old entries decode
with no run usage.
- apiserver: map run_usage into JobInfo; max_mem, avg_mem, used_mem,
cpu_time, peak_cpu, peak_cpu_time, peak_cpu_duration, mem_efficiency,
avg_cpu_efficiency, peak_cpu_efficiency become queryable columns.
- xcecli: -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK,
CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now
renders too), and a MEMORY USAGE / CPU USAGE block in jobs list -l.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Formatting only (gofmt -w on the 29 non-vendored, non-generated files
gofmt -l reported); swag-generated apiserver/docs/docs.go left as is.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
preCheckJobRequest returns (n, isArray, err): a bracketed job name is
parsed by ParseSubmitIndexExpr first, so name[1-N] and name[1] create
arrays (one-element arrays included) allocated with
generateArrayJobID, stored under the bare name; expression errors are
answered 400 with their specific message and create nothing; names
without brackets take the unchanged createArrayName path. The response
names the array (jobid = array ID, new array_size, "(n elements)"),
submit-time %I/%J use the first element, and the submit and dispatch
logs identify arrays by array ID. Fixes bug-1896 for REST and xcecli.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The apiserver's job store learned of a new job only from the
controller's asynchronous OnJobCreated / array push, so a 'jobs kill'
(or query) straight after 'jobs submit' could answer 'job not found'
for the job just accepted (2 of 4 immediate kills).
services.SubmitJob now records the accepted job right after the
controller's reply (which comes only after the job is committed and
applied): one record per element, base+i for arrays, stored with the new
DataStore.PutIfAbsent and marked JobInfo.Provisional (json:"-", not a
column). An OnJobCreated that already arrived is never overwritten; the
push that arrives later replaces the provisional record -- including in
dispatchJobArray, whose duplicate check no longer treats a provisional
element as already known.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scheduler-framework merge (3b6f53bb) dropped ac6131d's "pending_reason"
from models.JobInfo JOB_DEFAULT_COLUMN during conflict resolution (took
theirs). Default job queries returned pending_reason:null even though the
scheduler attaches reasons, the controller stores them on job->pend_reasons,
and the apiserver's in-memory store has them.
Found during the 2026-09-01 full job cluster verification: scheduler P1
TIME_WINDOW/HOST_UNAVAIL reasons confirmed end-to-end via controller
set_preason/apiserver hook debug logs and apiserver dataset-push debug logs;
only the default query column selection was missing. Verified after the fix:
night queue (window closed) -> main_reason:8 time_window; all workers stopped
-> main_reason:9 host_unavail with per-host details.
Co-Authored-By: Claude Code <noreply@anthropic.com>
After merging origin/scheduler-framework (scheduler-core), re-apply and
reconcile our fixes that Rongsong's branch lacked or implemented differently:
- handler.c: keep the code<0 dispatch-fail check (bug-1512: a code-0
success ack must NOT re-PEND, or dispatched jobs spawn duplicates).
Use sched_job_state_change (Rongsong's enqueue mechanism) instead of
sched_requeue_pend_job so the merged test framework (pop_jstate_change)
stays valid. Drop Rongsong's duplicate unconditional re-PEND case.
- scheduler.c: port sched_requeue_pend_job (bug-1494 class) adapted to
sched_get_job(&sched->core, ...) signature.
- types.h: re-add xce_sched_queue_t.dispatchwindow (our time-window PEND
reason field) dropped when taking Rongsong's types.h.
- models/jobs.go: re-add JobSubmitResponse.JobIDHuman (human job id).
- worker controller_connector.c: our bug-1509 resend_job_status on rejoin
success was already ported; add test stub.
- tests: adapt dispatch-fail test (code=0 → no re-PEND); fix sched_get_job
signature in white-box stubs.
Build passes (nix build .#xce). Unit + live-cluster verification pending.
Co-Authored-By: Claude <noreply@anthropic.com>
CreateJob now does FromHost: req.FromHost, a straight passthrough of
whatever xcecli (or any other client) sends in JobSubmitRequest.FromHost,
instead of deriving it server-side via ctx.ClientIP().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
GetJobsInfo/GetJobByIDs validated the -o/-l field list against
models.JobSubmitRequest{} -- the job-submission payload -- instead of
models.JobInfo{}, the query/response model (the already-correct
pattern used elsewhere in this file at line 1131). Response-only
fields like pending_reason don't exist on JobSubmitRequest, so any
such field made the whole listing fail outright with "unsupported
query field", not just omit that column.
core_limit/mem_limit exist on JobSubmitRequest but not on JobInfo --
switching wholesale would have newly broken those two previously-valid
fields. Added CoreLimit/MemLimit to JobAdditionalFields, the existing
merge-in bucket already used for pids/pgids/interactive_job_url.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
apiserver now returns pending_reason as an object ({main_reason,
main_reason_name, host_reasons, res_reasons}) since the PEND-reason work.
xcecli's QueryJob.PendingReason was a plain string, so every job list
failed with "cannot unmarshal object into string". Two fixes:
- xcecli/models/jobs.go: PendingReason is now a string-backed type with a
custom UnmarshalJSON that accepts null / empty string / plain string
(older API) / the object form (current API, rendered as readable text —
LSF bjobs -p style: "match_fail; host t1: match_fail; ..."). Unknown
objects fall back to the raw bytes instead of erroring, so a job list
can never fail on an unexpected pending_reason shape.
- apiserver/models/jobs.go: JobAdditionalFields gains PendingReason so the
fields= query (xcecli `jobs list -l`) no longer rejects pending_reason
with "unsupported query field".
- xcecli/common/output/page_print_jobs.go: adapt the verbose printer to the
new type.
Tests: xcecli/models/jobs_test.go — 6 cases (null / empty / plain string /
object / within QueryJob / unknown object). xcecli 9 packages + apiserver
models/controllers/utils all pass.
Verified on the test cluster: `xcecli jobs list` (no unmarshal error) and
`xcecli jobs list -l` show "PENDING REASONS: host_unavail; host t1:
host_unavail; host t2: match_fail; host t3: match_fail".
Co-Authored-By: Claude <noreply@anthropic.com>
The pending_reason field was declared on models.JobInfo but never populated:
scheduler computed PEND reasons and the controller stored them on
job->pend_reasons, yet the data stopped there. Two gaps closed:
1. controller/apiserver.c: apiserver_job_hook_fun now handles
HOOK_DETAIL_SETPREASON by pushing the updated job to API servers (reusing
apiserver_job_update_fun). Without this, a job whose PEND reasons change
never reached the API server — only JOB_NEW/UPDATE/START/EXIT events did,
and setpreason fired none of those.
2. backend/convert.go + convert_pure.go: cJobToModel / rawJobToModel now
extract xce_job_t.pend_reasons (main_reason + host/res reasons) and
marshal them into models.JobInfo.PendingReason as JSONB. models/jobs.go:
pending_reason added to JOB_DEFAULT_COLUMN so default job queries return
it (it was only in JSONBJFields, never selected).
Verified live on t1/t2/t3: a job with resreq "type == nosuchhost" stays
PEND and returns
"pending_reason":{"main_reason":0,"host_reasons":[{"host":"t1","reason":3},
{"host":"t2","reason":3},{"host":"t3","reason":3}]}
(main_reason=3 PREASON_MATCH_FAIL per host) via both /jobs list, /jobs/:id
and the SSE stream. 4 new unit tests (rawJobToModel pendReason marshal +
name mapping), all apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
The MCP/apiserver returned the raw 64-bit job ID (e.g. 73930964699447296)
while xce_test/xcecli display the dash form (e.g. 1-0-1786585485-0) that
parse_jobid accepts. A user taking the numeric form into xce_test fails to
parse it. The apiserver is the data-correctness owner (MCP only reads), so
it now emits the controller's fmt_jobid form alongside the numeric ID:
- models: JobSubmitResponse gains job_id_human; job list rows gain
job_id_human (processJobData), using the existing backend.FormatJobID.
- schedulerclient: URMJob.HumanID + SubmitJobResult.JobIDHuman parse it.
- Tests: backend.FormatJobID (real t1 job, pre-cluster1 job, array child);
processJobData adds job_id_human.
Also rename the SSE permission event:subscribe → event:view so mcp_readonly
stays strictly *:view (the read-only invariant test enforces).
Verified live on t1: scheduler_submit returns job_id_human 1-0-1786586474-0
and scheduler_get_job returns human_id.
Co-Authored-By: Claude <noreply@anthropic.com>
When a worker re-joins and already exists in workermgr, makeWorker_()
now overrides the existing slot instead of returning EEXIST. Admin-set
flags (LOCKEDU, CLOSED, HIDDEN, CLOSED_BY_ADMIN) survive the override.
Add XCE_WORKER_CLOSED_BY_ADMIN flag to distinguish admin-closed hosts
from workload-closed ones. worker_ctrl sets/clears it alongside CLOSED.
Refactor worker_init/worker_destroy into copy_worker/kill_worker helpers
shared by both fresh creation and override paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
