XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.
Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.
Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apiserver now returns pending_reason as an object ({main_reason,
main_reason_name, host_reasons, res_reasons}) since the PEND-reason work.
xcecli's QueryJob.PendingReason was a plain string, so every job list
failed with "cannot unmarshal object into string". Two fixes:
- xcecli/models/jobs.go: PendingReason is now a string-backed type with a
custom UnmarshalJSON that accepts null / empty string / plain string
(older API) / the object form (current API, rendered as readable text —
LSF bjobs -p style: "match_fail; host t1: match_fail; ..."). Unknown
objects fall back to the raw bytes instead of erroring, so a job list
can never fail on an unexpected pending_reason shape.
- apiserver/models/jobs.go: JobAdditionalFields gains PendingReason so the
fields= query (xcecli `jobs list -l`) no longer rejects pending_reason
with "unsupported query field".
- xcecli/common/output/page_print_jobs.go: adapt the verbose printer to the
new type.
Tests: xcecli/models/jobs_test.go — 6 cases (null / empty / plain string /
object / within QueryJob / unknown object). xcecli 9 packages + apiserver
models/controllers/utils all pass.
Verified on the test cluster: `xcecli jobs list` (no unmarshal error) and
`xcecli jobs list -l` show "PENDING REASONS: host_unavail; host t1:
host_unavail; host t2: match_fail; host t3: match_fail".
Co-Authored-By: Claude <noreply@anthropic.com>
Covers the case where the worker process stays alive and only the TCP link
to the controller drops (or the controller fails over): libxce's failover
loop reconnects transparently and xcecli_hello() returns 0, so
checking_connection() sees ret==0 and never takes its
xcecd_join_cluster()/resend_job_status() path — job state changes that
occurred while disconnected are never re-reported, leaving stale RUN/UNKWN
jobs on the controller.
- worker/controller_connector.c: new xcecd_reconnect_cb registered via
xcecli_set_reconnect_handler after a successful join, resending the actual
job state (bug-1509 semantics) on every transparent reconnect.
- lib/client.c (bug-1522): xcecli_start no longer resets rejoin_cb/rejoin_arg
to NULL — it is initialized by xcecli_ctx_init's calloc, and a pre-start
registration (scheduler sched_reconnect_cb, worker xcecd_reconnect_cb) must
survive first connect; the failover loop invokes it after a transparent
reconnect (xce_ctx_reconnect), which never re-runs xcecli_start. Previously
every pre-start registration was dead code.
- scheduler.c: move xcecli_set_reconnect_handler to AFTER a successful join —
xcecli_join also runs the failover loop, and a candidate-iteration rotate
(standby redirect) must not fire the rejoin hook before the session exists
(was firing sched_reconnect_cb during initial join → rejoin on
uninitialized state → invalid-opcode crash).
- test_libxce_retry.c: regression test for bug-1522
(test_reconnect_handler_survives_start_state_reset). Suite 296/296.
Functional verification on the test cluster: worker joins (host OK), runs a
job, and after a controller-link break re-joins ("Rejoined cluster
successfully after retry N"). Scheduler stays up (no invalid-opcode crash).
Co-Authored-By: Claude <noreply@anthropic.com>
Add queue dispatch-window support so a queue whose window is closed
(queues.yaml dispatch_window, e.g. 19:00-08:30) is treated as
non-electable and reports PREASON_TIME_WINDOW.
- lib/window.c: new xce_timewindow_is_open() — NULL/empty always open,
same-day and overnight (close < open) segments, day-scoped (0=every day,
1-7=Mon..Sun).
- scheduler: make_sched_queue duplicates dispatchwindow_config onto
xce_sched_queue_t.dispatchwindow (freed by kill_sched_queue);
sched_queue_is_ready() checks it; sched_queue_status_reason() now takes
the queue (was a bare status bitmask) and maps a closed window to
PREASON_TIME_WINDOW.
- fcfs: keep the windowed-out queue skip; do NOT attach the queue-status
reason to job->schedobj from the picker thread (bug-1521: races the
allocator thread's MOVE_PTR(obj->reason,...) and corrupts the heap).
- tests: test_window.c (5 cases) covers no-window/daily/overnight/
day-scoped/multi-seg; full suite 295/295 pass.
Cluster-verified: with a 09:00-17:00 window on the xce queue (currently
outside the window), fcfs reports "queue xce ... ready=0 reason=8"
(PREASON_TIME_WINDOW).
Co-Authored-By: Claude <noreply@anthropic.com>
Two bugs surfaced by the new scheduler_pend_reasons tool:
1. The apiserver returns job_status as the numeric XCE state bitmask
(PEND=0x01, RUN=0x02, EXIT=0x04, DONE=0x08), but getJobs read it with
rowStr (string), which returns "" for a number → every job mapped to URM
"unknown", so the pending/running status filter matched nothing.
→ JobStatusFromXCEState(int) + rowURMStatus fallback.
2. pending_reason is a JSON object in the row, not a string; rowStr returned
"" so Extensions["pending_reason"] was empty and the reason never parsed.
→ rowJSON(key) stores the raw JSON value; the MCP pend_reasons parser
handles the map form.
schedulerclient tests updated (JobStatusFromXCEState table). mcp-server +
schedulerclient test suites pass; live MCP call now returns the bjobs -p style
explanation for a PEND job.
Co-Authored-By: Claude <noreply@anthropic.com>
New read-only tool that answers "why is my job waiting": lists PEND jobs (or
one by id) and translates each job's pending_reason into a human-readable
scenario (LSF bjobs -p style). Maps the PREASON_* codes surfaced by the
PEND-reason work (4e2ac86): no_resource / match_fail / limit_global /
limit_host / queue_closed / queue_draining / time_window / host_unavail.
- internal/pendreason.go: pending_reason JSON parsing (mirrors apiserver
rawPendReason) + reasonScenario() translation + bjobs -p style summary.
- internal/app.go: HandlePendReasons — id param for one job, else pending
list filtered by queue/owner/limit.
- cmd/main.go: register scheduler_pend_reasons (read-only).
- 5 unit tests (known/unknown codes, parse from JSON, missing reason,
summary format). go test ./... + go vet clean.
Co-Authored-By: Claude <noreply@anthropic.com>
P0 (done, cluster-verified):
- main_reason was always 0: add_job_preason only set it when BOTH host and res
were NULL, but the allocator always attaches host/res detail rows. Now the
first detail add wins as main_reason, so the API's main_reason_name is
meaningful (was "unknown(0)"). Verified live: PEND jobs now report
main_reason=3/match_fail + per-host reason_name.
- limit plugin attach: host-based quota hit (hg1_slots_per_host) -> LIMIT_HOST
+ limit name; global quota hit (ug1_slots, ...) -> LIMIT_GLOBAL + limit name.
Previously both blocks were silent (job just stayed PEND).
- Extended the PREASON_* enum (was 3 codes) with LSF-style classes:
LIMIT_GLOBAL/LIMIT_HOST/QUEUE_CLOSED/QUEUE_DRAINING/TIME_WINDOW/HOST_UNAVAIL.
apiserver pendReasonName maps them; host/res detail rows now carry reason_name.
P1 (implemented):
- queue status: sched_queue_is_ready no longer a stub — closed/inactivated
queues are not electable; the fcfs election attaches QUEUE_CLOSED /
QUEUE_DRAINING to every PEND job in a skipped queue.
- worker filter: sched_try_alloc_on_host_ skips UNAVAIL/CLOSED workers and
attaches HOST_UNAVAIL instead of trying to allocate on them.
Unit tests: preason suite updated for the main_reason change + 3 new tests
(limit host/global, first-add-wins); 295 lib tests pass under ASAN.
apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
The apiserver stayed circuit-breaker-degraded after a full controller
restart until manually restarted. Two root causes:
1. Breaker recovery was decoupled from the breaker itself. The breaker can
be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while
the controller is electing) with no heartbeat ever failing, but
StartHeartbeat only recovered on a fail→success heartbeat transition
(prevFailed), so the breaker stayed degraded forever.
→ extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when
prevFailed OR the breaker is currently tripped. 5 new unit tests.
2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's
internal ctx state is XCE_CTX_ERR. After a controller restart whose
election outlasts the FAILOVER_CALL retry budget, the retry loop's
xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the
FF path never reconnects and the client stays disconnected forever.
→ lib: new xcecli_force_reconnect() forces a rotate+recover regardless
of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while
degraded.
Verified on cluster: after controller kill+restart, the apiserver logs
"backend re-joined controller after reconnect" and job submissions resume
without a manual restart; a fresh job runs to DONE. Controller, scheduler
(rejoin ok, bug-1513) and workers all auto-recover.
Co-Authored-By: Claude <noreply@anthropic.com>
The "libxce reconnect heap corruption" (ASAN bad-free, free(): invalid
pointer) was NOT a libxce bug — xce_ctx_reconnect is fine. The crashes are
in the scheduler's rejoin state re-sync path (config/dataset re-push),
fixed with the reconnect/rejoin feature now integrated:
1. group.c makeGroup refresh: FREEUP(m->u.name) raw-freed shared interned
xstrings (makeString is a global refcounted table); the correct release
is dereference(), matching group_killer. → dereference().
2. limit.c limit_update_config: after key_add returns -EEXIST (rejoin
re-push — limit name already present) it called kill_limit(limit), but
xkvmgr_add's vclear already freed the limit via the kvmgr datakiller
(=kill_limit). → skip on -EEXIST (idempotent re-push).
3. handler.c/scheduler.c: the config push was applied inline on the vcpu
thread (sched_init_config adds queues/plugin queues) while job updates
flow through pend_updates on the main loop — concurrent rejoin re-push
let fcfs_update_job see a NULL j->queue (SIGSEGV). Route the config
push through pend_updates (XCE_SCHED_OP_UPDATE_CONFIG, killer case
already existed) and process config requests FIRST in sched_update via
a staging queue so jobs always see their queues/resources.
Verified: 295 lib unit tests pass under ASAN (incl. 2 new group-refresh
regression tests), scheduler survives 3 consecutive controller restarts
(each rejoin ok), full job lifecycle (submit→dispatch→DONE) on the
cluster. Also fixed stale build deps (libxce_deps missing librtti_dep,
scheduler_deps missing ssl_dep/c_capnproto_dep, unity include path).
Co-Authored-By: Claude <noreply@anthropic.com>
component-failover-adoption Req 4: when the apiserver's controller connection
is down, backend-requiring API requests must return a clear service-unavailable
error rather than hanging. Adds:
- health.go: an atomic circuit breaker (IsDegraded/MarkDegraded/MarkHealthy).
- callBackend: the four write calls (jobsubmit/jobctrl/queuectrl/hostctrl)
now route through it — degraded ⇒ immediate "backend controller unavailable"
error without invoking the lib; a transport OR XCE not-ready/masterless/
standby failure (E_XCE_NOMASTER/REDIRECT/NOT_READY/STANDBY) trips the breaker
so the NEXT call fails fast. Semantic rejections (-EINVAL/-ENOENT/E_XCE_BAD_*)
never trip it.
- StartHeartbeat: a failed tick marks degraded; recovery re-joins (rejoinBackend)
and marks healthy.
Verified live (t1 cluster): kill controller → first submit returns
xcecli_jobsubmit failed: -10013 (bounded, no hang), second submit returns
"backend controller unavailable (reconnecting)" in ~11ms. 7 unit tests.
Recovery-via-heartbeat after a FULL controller death is NOT yet reliable — the
lib transport reconnect does not always re-establish to the restarted
controller (same class as the scheduler reconnect; tracked under task 13).
Co-Authored-By: Claude <noreply@anthropic.com>
Closes the G-5 test gap mirroring the G-8 routing test: the rusage notify
handler enqueues XCE_SCHED_OP_JOB_RUSAGE and sched_update() must dispatch it to
sched_apply_job_rusage(), storing the report on the matching job (bug-765 G-5
remediation, production fix already in 93124c5). Driving the real sched_update()
dequeue loop with a recording stub, asserts the request reaches
sched_apply_job_rusage and the job's rusages field is set.
11/11 sched_redispatch_test PASS; full scheduler suite 6/6 OK. Live: job ran to
DONE on the deployed scheduler (G-3 leak fix in place), zero ASAN errors.
Co-Authored-By: Claude <noreply@anthropic.com>
component-failover-adoption Req 2 (partial): two of the three daemons now
self-heal after a controller restart without an operator restart.
- worker (controller_connector.c): checking_connection() previously gave up
after 5 retries and called handle_disconnect() (which set running=false and
exited the connector thread). It now retries indefinitely — the connector
thread is dedicated to this loop so the unbounded retry/sleep cannot block
the worker's other liveness duties. Verified live: killing the controller
leaves all 3 workers alive and retrying; they re-join when it returns.
- apiserver (clib.go): xcecli_heartbeat is fire-and-forget — it best-effort
reconnects the transport but never re-joins, so after a controller restart
push notifications stop. StartHeartbeat now tracks heartbeat failure→success
and, on recovery, calls xce_backend_rejoin (xcecli_join as HostRole_apiserver)
to re-register and resume pushes. Verified live: "backend re-joined
controller after reconnect" logged during a controller kill/restart.
Scheduler reconnect NOT included: the rejoin works but exposes a pre-existing
libxce race — xce_ctx_reconnect's endpoint_fini tears down the old endpoint
while a lib vtask/io thread is still processing it, a deterministic double-free
(bad-free in the loader, same freed address offset across runs). Root-cause in
libxce netio/endpoint lifecycle is a separate task (documented in
left_scheduler_work.md).
Co-Authored-By: Claude <noreply@anthropic.com>
Closes the msg-gap G-8 test gap the research flagged ("handler→dequeue ROUTING
is verified by code review, NOT an automated integration test"). The
remediation (UPDATE_JOB_FIELDS op routed to sched_update_existing_job) is
already in the code (93124c5); these tests lock the routing:
- test_update_fields_routes_to_inplace_path: driving the real sched_update()
dequeue loop, an UPDATE_JOB_FIELDS request reaches sched_update_existing_job
and mutates the existing job in place (same object, runtime state kept).
- test_bulk_update_existing_job_no_damage: a re-pushed dataset row for an
existing jobid is EEXIST-dropped by key_add (via the real sched_add_job) —
the live job keeps identity/state/runtime state.
10/10 sched_redispatch_test PASS; full scheduler suite 6/6 OK. Live cluster:
job PEND→RUN→DONE via notify updates, no WARN, no duplicate instance.
Co-Authored-By: Claude <noreply@anthropic.com>
The pending_reason field was declared on models.JobInfo but never populated:
scheduler computed PEND reasons and the controller stored them on
job->pend_reasons, yet the data stopped there. Two gaps closed:
1. controller/apiserver.c: apiserver_job_hook_fun now handles
HOOK_DETAIL_SETPREASON by pushing the updated job to API servers (reusing
apiserver_job_update_fun). Without this, a job whose PEND reasons change
never reached the API server — only JOB_NEW/UPDATE/START/EXIT events did,
and setpreason fired none of those.
2. backend/convert.go + convert_pure.go: cJobToModel / rawJobToModel now
extract xce_job_t.pend_reasons (main_reason + host/res reasons) and
marshal them into models.JobInfo.PendingReason as JSONB. models/jobs.go:
pending_reason added to JOB_DEFAULT_COLUMN so default job queries return
it (it was only in JSONBJFields, never selected).
Verified live on t1/t2/t3: a job with resreq "type == nosuchhost" stays
PEND and returns
"pending_reason":{"main_reason":0,"host_reasons":[{"host":"t1","reason":3},
{"host":"t2","reason":3},{"host":"t3","reason":3}]}
(main_reason=3 PREASON_MATCH_FAIL per host) via both /jobs list, /jobs/:id
and the SSE stream. 4 new unit tests (rawJobToModel pendReason marshal +
name mapping), all apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
Simplification per review feedback on e3b3ffa: a generic schema-driven
unit field touched the capnp wire schema (2 structs), C structs
(xce_resource_t/xce_resval_t), resobj.c's deep-copy/destroy, opget.c's
reply-build lookup, resources.yaml/resources.schema.json, and mapping.py
-- real cross-cutting cost for a mechanism only one resource (maxmem)
actually needs today.
Reverted all of that. The controller CLI (test.c) now just knows
"maxmem" is bytes-valued via a small explicit is_bytes_valued_resource()
name list, and adaptively picks the best-fit human unit at display time
(format_bytes_human, unchanged) -- same observable behavior (resval/
workerlist render maxmem as e.g. "30.90 GB"), zero wire schema surface.
Regenerated the xceadmin golden fixture back to its pre-unit-field size
(5304 bytes, exactly matching the size right after the mxj-only fix,
confirming a clean, complete removal).
Verified: 1095/1095 controller_unit_test (3 runs; one hit the
pre-existing flaky segfault already logged as bug-1553, unrelated),
228/228 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three bugs found while diagnosing a live controller log, all in the
worker resource-value path:
1. Every host relying on the documented hosts.yaml auto-detect default
(no max_slot, meaning "use CPU count") got 0 scheduling slots,
permanently blocking the scheduler from placing jobs on it. mxj is
UInt32 on the wire and can't carry the schema's -1 sentinel; mapping.py
worked around this by leaving it unset, which defaults to 0 -- and 0 is
never < 0, so apply_join_state()'s auto-detect fallback never fired.
Fixed by encoding the sentinel as 0xFFFFFFFF, which the controller's
`int mxj` field reinterprets back to -1 via its existing implicit
uint32_t->int conversion. No C-side or wire schema change needed for
this part; the fallback itself was already correct but had zero test
coverage for a negative mxj, now added.
2. update_resvals()/update_resval_list() logged the failure reason via
PERROR, which reads process-global errno -- but update_resval()'s
failure paths return raw negative codes directly and never touch
errno, so the controller log filled with misleading "update_resval:
Success" / "update_resval: Resource temporarily unavailable" lines
instead of the real reason. Fixed to decode the actual rc via
xce_strerror_r and name the failing resource.
3. maxmem is canonically stored/wired as raw bytes but had no unit
metadata, so resval/workerlist printed it as a bare number that read
as garbage (e.g. workerlist's fixed-width columns visually smeared a
multi-GB byte count across the MaxCPU/MaxMem headers). Added an
optional `unit` field to the Resource/ResourceVal capnp structs (and
xce_resource_t/xce_resval_t/resources.yaml) so resval and workerlist
can render a human-readable value (e.g. "30.90 GB") while the stored
value stays raw bytes -- display-only, every other resource unaffected.
Regenerated the xceadmin golden fixture (fixtures/full/expected.bin)
twice: once for the mxj sentinel encoding change, once for the new
Resource.unit field growing every resource's wire size by one pointer
slot. Both diffs verified byte-for-byte isolated to exactly the expected
change before regenerating.
Verified: 1095/1095 controller_unit_test (stable across repeated runs),
229/229 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Two related findings logged to .wolf/buglog.json but deliberately not
fixed here (out of scope per discussion): a pre-existing flaky segfault
in the combined controller_unit_test binary, and maxswap/maxtmp being
hardcoded stub values on the worker side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The controller acknowledges EVERY dispatched job through
NotifyData_data_jobDispatchFail (controller/job.c handle_jobdispatch):
a negative code is a real rejection, code 0 is the success ack (the else
branch calls notify_dispatch(jobid, 0)). sched_handle_notify treated any
jobDispatchFail as a failure and returned the job to PEND, so every
successfully dispatched job was immediately re-PENDed, re-elected, and
re-dispatched while the original instance kept running — spawning
duplicate instances cluster-wide. On a 3-worker cluster 24 submitted jobs
ran 38 instances and piled onto the first worker (t1) because its slots
were counted for each duplicate.
Fix: only sched_requeue_pend_job() when fail->code < 0; consume the
success ack (code 0) as a no-op — the job is already RUN
(sched_commit_decision marks it at commit). This was a regression of the
bug-1494 re-PEND logic (bug-1510 fix) that lacked the code guard.
Verified live on t1/t2/t3: 24x sleep-300 jobs distributed exactly 8/8/8,
zero duplicates, all DONE. Add white-box test
test_handle_dispatch_fail_requeues_only_on_negative_code.
Co-Authored-By: Claude <noreply@anthropic.com>
The MCP/apiserver returned the raw 64-bit job ID (e.g. 73930964699447296)
while xce_test/xcecli display the dash form (e.g. 1-0-1786585485-0) that
parse_jobid accepts. A user taking the numeric form into xce_test fails to
parse it. The apiserver is the data-correctness owner (MCP only reads), so
it now emits the controller's fmt_jobid form alongside the numeric ID:
- models: JobSubmitResponse gains job_id_human; job list rows gain
job_id_human (processJobData), using the existing backend.FormatJobID.
- schedulerclient: URMJob.HumanID + SubmitJobResult.JobIDHuman parse it.
- Tests: backend.FormatJobID (real t1 job, pre-cluster1 job, array child);
processJobData adds job_id_human.
Also rename the SSE permission event:subscribe → event:view so mcp_readonly
stays strictly *:view (the read-only invariant test enforces).
Verified live on t1: scheduler_submit returns job_id_human 1-0-1786586474-0
and scheduler_get_job returns human_id.
Co-Authored-By: Claude <noreply@anthropic.com>
Audit of apiserver's three task-4.6 coverage areas (degraded-to-healthy
transitions, concurrent-request safety, candidate-list parity):
degraded-to-healthy and candidate-list parity were already fully
satisfied by tasks 4.1/4.5's own tests. Found one genuine gap: reconnect.go's
needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access
test coverage despite being reached from 5 real concurrent HTTP-wrapper
call sites plus the heartbeat goroutine -- health.go's existing
50-goroutine stress test only covered the separate degraded-until
timestamp. One new test closes it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds an atomic degraded-until state (health.go: IsDegraded/MarkDegraded/
MarkHealthy) that lets request handlers fail fast with a clear
unavailable error instead of blocking on xcecli's own ~100s retry
budget when the backend is known-degraded.
Consolidates what tasks 4.3/4.4 left as duplicated inline
"call xcecli_*; checkAndPerformRejoin()" pairs across five wrapper
functions into one shared callBackend() helper: checks IsDegraded()
before every call (fail fast, skip the C call and the rejoin check
entirely on the degraded path -- checking rejoin there would itself
block up to ~100s, defeating the fail-fast purpose), marks degraded on
-E_XCE_NOT_READY, marks healthy at checkAndPerformRejoin()'s existing
success point (the single call site both the write path and the
idle-period heartbeat path already funnel through).
The circuit breaker's own TTL-based state is deliberately never read
by the rejoin-retry mechanism -- checkAndPerformRejoin() has no
IsDegraded() dependency, so a fast-fail window never suppresses
retry-until-actually-joined; heartbeat.go's unconditional per-tick
call keeps that cadence alive regardless of degraded state, and a
10s TTL (well under the ~100s retry budget) lets write traffic resume
contributing once it clears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
xcecli_heartbeat() is XCECLI_FF_CALL-wrapped -- a deliberately
non-retrying fire-and-forget path that does its own best-effort
one-shot reconnect but never calls xcecli_invoke_rejoin() (that only
fires from the write-call retry path). Left as-is, a controller
failover with no in-flight write request would silently repair the
transport connection without ever re-joining, permanently starving
push notifications.
runHeartbeat() gains two callback hooks fired at the right points --
onFailureToSuccess on a genuine transition, onSuccessTick on every
successful tick thereafter -- wired directly to task 4.3's existing
setNeedsRejoin()/checkAndPerformRejoin() from StartHeartbeat(). No
second rejoin mechanism: checkAndPerformRejoin() is a cheap no-op when
nothing is pending, so calling it unconditionally on every successful
tick is safe and matches the requirement that retry isn't gated on the
transition edge alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Registers a rejoin flag-setter (go_backend_rejoin_handler) as the
client library's reconnect callback: zero I/O, sets an atomic flag,
returns immediately -- avoiding self-deadlock on the library's
per-ctx failover lock the same way worker's and scheduler's rejoin
callbacks already do. The C shim, its extern declaration, and the
xcecli_set_reconnect_handler() registration call all live together in
clib.go's xce_backend_init(), placed after xcecli_start()'s success
check (xcecli_start() wipes any earlier-registered handler as part of
its own init -- the same ordering bug found and fixed in worker's/
scheduler's task 2.3/3.1 work).
A shared checkAndPerformRejoin() helper wraps every gCtx.ctx-based
backend call (SubmitJob, ControlJob, ControlQueue, ControlHost,
FetchFile): after the call returns, if a rejoin is pending, issue
xcecli_join() as a separate step and clear the flag only on success --
on failure it stays set so the next backend call's return retries it,
no separate retry loop.
needsRejoin is a proper sync/atomic flag rather than a plain bool,
since apiserver (unlike the single-threaded C daemons) serves
concurrent HTTP requests that can all reach the shared helper at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Add four tools closing the submit→watch→analyze→suggest loop in the
request/response MCP model:
- scheduler_wait_job: poll a job to a terminal state or timeout.
- scheduler_analyze_failure: gather job record + output logs for failure
analysis (interpretation left to the LLM).
- scheduler_suggest_resources: conservative processor/node suggestions from
current cluster availability.
- scheduler_estimate_wait_time: rough wait estimate from pending count and
free cores (guidance, not a guarantee).
All verified live on t1 against the running apiserver.
Co-Authored-By: Claude <noreply@anthropic.com>
Add WatchJobs(ctx) to the Client interface and HTTPClient: opens the
apiserver's /xce/v1/events SSE stream and yields JobEvent values until the
context is cancelled or the stream ends. JobEvent carries the event type,
job id, status string, exit code, and a raw job snapshot.
Also fix ListJobs' filter params: the apiserver accepts filters via the
"filter" query param in field:op:value form (e.g. queue_name:eq:normal);
the individual queue_name/submit_user params it ignores with a 400, which
broke every filtered list. Multiple conditions are brace-wrapped {a},{b}.
Verified live on t1: WatchJobs captured job.updated PEND→RUN→DONE for a
submitted job.
Co-Authored-By: Claude <noreply@anthropic.com>
Add a Server-Sent Events endpoint /xce/v1/events streaming job lifecycle
events (job.created/updated/deleted/finished) to subscribers. The
DefaultHandler now publishes every job event to a process-wide EventBroker;
the SSE handler subscribes and streams JSON frames with a keep-alive comment
during idle.
- backend/broker.go: thread-safe fan-out Broker (drops for slow consumers so
the notification path never blocks), Event type with job_status string.
- backend/handler.go: OnJob* publish to DefaultBroker.
- controllers/events.go: SSE handler (gin, text/event-stream, keep-alive).
- router + whitelist + rbac: /xce/v1/events route, event:subscribe
permission granted to mcp_readonly/mcp_operator/cluster_admin.
- Tests: broker fan-out/slow-consumer/unsubscribe, handler publishes, SSE
end-to-end via httptest.
Verified live on t1: SSE stream delivered job.updated PEND→RUN events for a
submitted job.
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces apiserver's two connect mechanisms -- a per-peer Go loop that
hardcoded nmasters=1 per attempt, and a separate direct-construction
path for the legacy single-host config -- with one pipeline:
buildClusterConfig() parses (via task 4.1's C parser) or synthesizes a
single-candidate config, validates it, then ConfigConverter
(xce_cluster_config_to_client_config) produces the client config for
one xce_backend_init() call. A 3-candidate config now actually
exercises the C library's own failover/standby-redirect handling on
initial connect, matching worker's and scheduler's task 2.1/3.1 work,
instead of a Go-level per-candidate retry loop that never built a real
multi-master config.
Along the way: found and fixed a genuine use-after-free. The deleted
tryConnect() freed its C.CString-allocated host address immediately
after its single connect call returned, but xcecli_ctx_init() stores a
raw pointer to the config forever (client.c:38) and every later
failover rotation re-dereferences it (client_failover.c) -- so any
reconnect after the first would read freed memory. Fixed by giving the
cluster/client config structs package-level, process-lifetime storage
(gClusterCfg/gClientCfg), mirroring worker's and scheduler's file-scope
static pattern, with host/from/key written into fixed-size C arrays
instead of freed heap pointers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's hand-written Go parsePeer() with a CGO wrapper
around the shared C parser+validator pair (xce_parse_peers then
xce_cluster_config_validate, both src/lib/cluster.c) that worker and
scheduler already use from their own main.c's. Gives apiserver the same
[name=]host:port[,...] syntax (including the name= prefix the old Go
parser could not express) and identical malformed-candidate rejection
(empty host, port <= 0, non-numeric port) -- round-1 review caught the
first pass skipping the validate() half, silently accepting inputs
worker/scheduler both reject.
A malformed entry now rejects the whole candidate list rather than
being skipped with a warning (matches xce_parse_peers()'s own
all-or-nothing construction and worker/scheduler's precedent) --
surfaced loudly via RegisterToBackend's error, not silently.
Parsing only -- RegisterToBackend()'s connect loop and tryConnect()
are unchanged; collapsing them into a single connect call is task
4.2's job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Proves task 3.2's retry-until-success-or-shutdown wrapper's guarantees
hold across multiple queued decisions drained through the real
xce_sched_driver_on_decision(), not just a single isolated call:
decisions queued behind a stuck one publish afterward in original
order (Req 7 AC3), and shutdown mid-retry costs at most one retry
cycle regardless of queue depth -- decisions behind the stuck one see
core.stop already true and are never attempted (Req 7 AC2). Also adds
a source-scan confirming this stays scoped to the scheduler's own
publish direction (Req 7 AC5) and a fresh-instance test confirming a
restart still loses in-flight queued decisions, unchanged by this work
(Req 7 AC6).
Pure test-coverage task -- zero production changes, matching the
review's independently-reproduced RED->GREEN evidence that the
existing task-3.2 implementation already satisfies these guarantees.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds a worst-case-occupancy test proving the scheduler's picker/allocator
pipeline buffers briefly past a stalled publish (decisions cbuf cap 5,
jbuf cap 1) then genuinely blocks until connectivity recovers, with zero
job loss (Req 3 AC2). Adds a source-scan regression guard confirming the
scheduler makes no periodic heartbeat call of its own (Req 3 AC3).
Also fixes a real gap found along the way: sched_init()'s
all-candidates-unreachable and join-failure log lines named only a
candidate count, never which candidates were attempted (same class of
bug worker's task 2.3 fixed). Added sched_format_candidate_list(),
covered by a real dup2()-based stdout capture around a live sched_init()
failure path (round-1 review caught this diagnostic shipping with zero
test coverage; fixed in round 2).
Multi-candidate join and kill-primary/reconnect were validated manually
against a live 3-node cluster rather than committed as an automated
test, following worker's task 2.3 precedent -- no automated live-deploy
E2E harness exists for any C daemon in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Wraps sched_publish_dispatch()/sched_publish_pend_reason() in a shared
retry loop that keeps retrying a -E_XCE_NOT_READY-exhausted publish
until it succeeds or the scheduler is shutting down (core.stop checked
before each attempt, not only after), composed with the pending-rejoin
follow-up established in task 3.1. Extended-retry success logs
distinctly from an ordinary first-attempt success (Req 7 AC4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replace scheduler's inline per-candidate loop and separate single-host
branch with one unified pipeline (mirrors worker's task 2.1 fix): build
exactly one validated candidate list, whether from --peers or synthesized
as a single candidate from the legacy default, then connect through the
shared xce_cluster_config_to_client_config() helper exactly once.
Adds sched_on_reconnect(), a rejoin callback that only sets a flag and
returns immediately (no I/O, no xcecli_* calls) to avoid the self-deadlock
worker hit in task 2.3 — registered via xcecli_set_reconnect_handler()
after xcecli_start(), since xcecli_start() resets any handler registered
earlier back to NULL.
Adds test_scheduler_connect.c: behavioral tests confirm no-peers and
single-candidate configs produce identical connect log sequences and that
unreachable candidates fail fast without retry loops; structural tests
source-scan scheduler.c to confirm a single xcecli_ctx_init() call site, a
single xcecli_join() call site, and a rejoin callback body free of I/O and
xcecli_* calls.
Live-validated against a 3-node HA controller cluster: a scheduler
configured with the standby listed first in --peers fails over via
xcecli's built-in retry-next mechanism to the primary and joins
successfully.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Confirm connector thread is isolated from worker_process_update() (main
thread) and resource_sync_thread() (own thread) via code read plus a real
pthread concurrency test. Add unit and live 3-node E2E coverage for
candidate/reconnect behavior, all-candidates-unreachable diagnostics, and
liveness cadence under concurrent reconnect.
The E2E run surfaced two production bugs the 2.1/2.2 unit tests missed:
xcecli_start() resets the reconnect callback to NULL after it runs, so
registering it before xcecli_start() silently no-ops the whole rejoin
mechanism; and a successful worker join returns its assigned slot index
as reply.status (any rc >= 0), not always exactly 0, so checking_connection()'s
rc == 0 success check never fired on a real non-zero-slot join. Both fixed,
plus per-candidate names added to the all-unreachable failure log.
Root-check bypass needed for the E2E run is gated at compile time via the
existing VIRTUAL_CLUSTER flag, not a runtime env var, so the real xcecd
binary's root requirement has zero bypass surface.
Thread cluster_cfg by value through xcecd_config_t -> xcecd_context_t ->
cc_ctx so it's populated before controller_connector_start() fires the
first join, instead of after xcecd_start() returns. checking_connection()
now retries a failed post-reconnect xcecli_join() on the next liveness
probe instead of leaving needs_rejoin cleared on mere attempt.
When a worker is re-pushed (e.g. it rejoined with fresh resvals after the
controller/worker had been down), sched_update_workers built a new worker and
sched_add_worker's key_add returned EEXIST — the fresh worker was dropped and
the scheduler kept the stale state (slots=0 from the dead/absent period).
PEND jobs never dispatched even after the worker returned with full capacity
(bug-1494, controller-restart self-heal).
sched_update_workers now updates an already-known worker in place: swap the
fresh resvals/resInUse/status into the existing worker (the stale arrays are
freed when the discarded object is torn down) and adopt its host_eval_ctx.
Also NULL-guard make_sched_worker (it can return NULL).
Verified on the t1 cluster: scheduler started while the worker was dead
(slots=0), worker rejoined (slots=8), previously-PEND jobs dispatched and
completed, no ASAN.
Co-Authored-By: Claude <noreply@anthropic.com>
Replace worker's separate default/single-host and multi-candidate connect
branches with one path: synthesize-or-accept a candidate list, validate,
convert via xce_cluster_config_to_client_config(), connect once. Register
a flag-only rejoin callback (no I/O, no client-lib reentry) and remove the
liveness probe's bounded give-up so the library owns retry/recovery.
xce_cluster_config_to_client_config() converts a validated candidate
list into a fully-populated xce_client_config_t, copying candidates
in order with no special-casing for a single candidate. The
truncate-with-warning safety net stays in place as a defensive guard
now that N_MAX_MASTERS and MAX_HA_CANDIDATES match.
Also removes a WARN check in xce_cluster_config_validate() that
became unreachable dead code once task 1.1 reconciled the two
candidate-count constants (both checks tested the same field against
the same constant with no mutation between them).
Task 1.2 of component-failover-adoption (Requirement 1 AC1-AC7).
N_MAX_MASTERS (client.h) now derives from config.h's MAX_HA_CANDIDATES
instead of an independently-maintained literal (3 vs 4 previously).
Fixes worker's and scheduler's hardcoded `> 3` startup checks to
reference the constant symbolically, and updates the client-retry
test fixture so it doesn't exercise an uninitialized 4th slot.
Task 1.1 of component-failover-adoption (Requirement 1 AC7).
The controller rejects a dispatch when the chosen worker is not in a
dispatchable state (UNAVAIL after a worker death, or still joining) and
replies with a jobDispatchFail notify. The scheduler had no handler for that
event, so a job it had just committed (state=RUN) was never returned to PEND
— it was never re-elected and stayed PEND cluster-wide even after the worker
recovered.
Handle NotifyData_data_jobDispatchFail: log the failure and call the new
sched_requeue_pend_job(), which frees the dispatch bookkeeping, resets the
job to PEND, and re-enqueues it in the fcfs pendJobs queue so the next
election session re-dispatches it. White-box test TUs that #include
handler.c get a sched_requeue_pend_job stub.
Co-Authored-By: Claude <noreply@anthropic.com>
Merge the scheduler-join NULL guard from the remote js branch: if the
controller's cluster config is not loaded yet (cold boot / config not
pushed yet), xcesched_push_configs would dereference a NULL cluster_config
and crash the controller (ASan SEGV on scheduler join). Skip the push; the
scheduler receives config on a later join once config is available.
Co-Authored-By: Claude <noreply@anthropic.com>
- Add OPCode_probeConfigTag dispatch in cluster.c
- Implement probe_config_tag handler returning config tag via
standard xce_reply() → RequestReply envelope
- Populate redirectHost/redirectPort for follow_redirects support
- Wire probeConfigTag reply through typemap
- Add probeConfigTagReply struct to controller.capnp schema
- probe_config() now wraps probe in Request envelope matching push_config()
- Controller OPCode_probeConfigTag handler replies through xce_reply()
with redirectHost/redirectPort populated
- follow_redirects works identically for both push and probe flows
- Change default record_dir from /var/log/xce/ to /tmp (always exists)
- Add mkdir(record_dir, 0755) at startup for custom --record-dir paths
- Split validation_error_text into 6 distinct per-errno messages
- Accept human-readable job IDs in 'info job' via parse_jobid()
- Add 'info jobs' command to list all jobs in sim REPL
- Format job IDs with fmt_jobid() for xce_test-consistent output
- Replace printf with fprintf(stdout, ...) in all REPL files
to bypass -Wl,--wrap=printf linker interception
The "Deploying the Scheduler" section predated scheduler-core and only
covered normal-mode startup. Adds session recording (--record-dir,
xceadmin scheduler record start|stop|status, filename validation),
xceadmin scheduler dump for offline .replay inspection, and simulation
mode (--sim, --sim-speed, --sim-verify, --sim-alt-events, --sim-output,
--interactive REPL) — all verified against the actual argp help output
and xceadmin cli.py source, not assumed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
