Full job cluster verification on 2026-09-01 (HEAD a34p5yk) confirmed the
P1 reason restore end-to-end: TIME_WINDOW (night queue, window closed) and
HOST_UNAVAIL (all workers stopped) both surface via the jobs API; worker
restart without scheduler restart recovers dispatch (bug-1511 concern
disproven). Also records the third scheduler-framework merge regression
(pending_reason dropped from JOB_DEFAULT_COLUMN, fixed in 070e9992) and the
apiserver OPCode_notify-not-delivered note.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The scheduler-framework merge (2026-08-25) silently regressed the PEND
reason P1 work (4e2ac86/ee79179): the old fcfs.c became dead-on-default
(builtin_fcfs replaced it), resource.c/limit.c conflict resolution took
Rongsong's side, and the merge-fix only re-added the dispatchwindow FIELD
without the logic. After the merge, sched_queue_is_ready/sched_job_is_ready
were `return 1` stubs and only MATCH_FAIL/NO_RESOURCE were ever emitted.
Restore P1 on the active (framework) path, keeping every reason attach in
the allocator thread (bug-1521: the picker thread must not write
job->schedobj->reason — it races the allocator's MOVE_PTR):
- core.c: sched_job_is_ready (PEND & not suspended) and sched_queue_is_ready
(QUEUE_STAT_CLOSED/INACT_ADMIN/INACT_WINDOW gate) real implementations.
- scheduler.c: allocator loop injects sched_queue_status_reason() — queue
status first, then the dispatch window evaluated against the core's
virtual clock (xce_core_now/1e9) so live and sim modes agree; attaches
QUEUE_CLOSED/DRAINING/TIME_WINDOW and pends the job. Adds <time.h>.
- resource.c: skip UNAVAIL/CLOSED/CLOSED_BY_ADMIN workers + HOST_UNAVAIL
(restored from 4e2ac86).
- limit.c: attach LIMIT_GLOBAL/LIMIT_HOST + carry reason in
limit_job_election (restored from 4e2ac86).
Two further merge regressions found during cluster verification (the
framework core path never carried the queue dispatch window):
- core_events.c core_build_queue: duplicate q->dispatchwindow_config onto
xce_sched_queue_t.dispatchwindow (make_sched_queue did this; the core path
dropped it, so TIME_WINDOW could never fire on the default path).
- core_events.c core_upsert_queue update branch: refresh dispatchwindow on
QUEUE_UPDATE.
Regression test: test_core_ready_gates.c (white-box #include of core.c,
7 cases covering both gates), registered in scheduler testsuite meson.build.
Cluster-verified on t1/t2/t3 (nix xcecore closure + AppImage standard
deployment): night queue (window closed) -> main_reason=8 time_window;
all workers down -> main_reason=9 host_unavail with per-host detail;
closed queue -> job stays PEND; normal queue baseline -> DONE. Scheduler
suite 52/53 (1 pre-existing admin-request symlink-escape failure, unrelated).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Expose two backend capabilities MCP previously omitted:
- scheduler_checkpoint: control action "checkpoint" (CRIU)
- scheduler_restart: POST /xce/v1/jobs/:id/restart (re-queues as a new
job, resuming from the last checkpoint if one exists; returns new jobid)
schedulerclient gains CheckpointJob/RestartJob (RestartJobResult,
JobRestartRequest); Client interface updated. Both tools marked
destructive (confirm required). tools/list now reports 17 tools.
Unit tests added/updated in both modules — all pass. Backend endpoints
are existing apiserver routes; no backend changes in this commit.
Also syncs left_scheduler_work.md §八.4 (checkpoint/restart done; the
only remaining MCP tool gap is scheduler_modify, blocked on a controller
job-field-update opcode that does not exist yet).
Co-Authored-By: Claude Code <noreply@anthropic.com>
The apiserver stayed circuit-breaker-degraded after a full controller
restart until manually restarted. Two root causes:
1. Breaker recovery was decoupled from the breaker itself. The breaker can
be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while
the controller is electing) with no heartbeat ever failing, but
StartHeartbeat only recovered on a fail→success heartbeat transition
(prevFailed), so the breaker stayed degraded forever.
→ extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when
prevFailed OR the breaker is currently tripped. 5 new unit tests.
2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's
internal ctx state is XCE_CTX_ERR. After a controller restart whose
election outlasts the FAILOVER_CALL retry budget, the retry loop's
xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the
FF path never reconnects and the client stays disconnected forever.
→ lib: new xcecli_force_reconnect() forces a rotate+recover regardless
of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while
degraded.
Verified on cluster: after controller kill+restart, the apiserver logs
"backend re-joined controller after reconnect" and job submissions resume
without a manual restart; a fresh job runs to DONE. Controller, scheduler
(rejoin ok, bug-1513) and workers all auto-recover.
Co-Authored-By: Claude <noreply@anthropic.com>
The "libxce reconnect heap corruption" (ASAN bad-free, free(): invalid
pointer) was NOT a libxce bug — xce_ctx_reconnect is fine. The crashes are
in the scheduler's rejoin state re-sync path (config/dataset re-push),
fixed with the reconnect/rejoin feature now integrated:
1. group.c makeGroup refresh: FREEUP(m->u.name) raw-freed shared interned
xstrings (makeString is a global refcounted table); the correct release
is dereference(), matching group_killer. → dereference().
2. limit.c limit_update_config: after key_add returns -EEXIST (rejoin
re-push — limit name already present) it called kill_limit(limit), but
xkvmgr_add's vclear already freed the limit via the kvmgr datakiller
(=kill_limit). → skip on -EEXIST (idempotent re-push).
3. handler.c/scheduler.c: the config push was applied inline on the vcpu
thread (sched_init_config adds queues/plugin queues) while job updates
flow through pend_updates on the main loop — concurrent rejoin re-push
let fcfs_update_job see a NULL j->queue (SIGSEGV). Route the config
push through pend_updates (XCE_SCHED_OP_UPDATE_CONFIG, killer case
already existed) and process config requests FIRST in sched_update via
a staging queue so jobs always see their queues/resources.
Verified: 295 lib unit tests pass under ASAN (incl. 2 new group-refresh
regression tests), scheduler survives 3 consecutive controller restarts
(each rejoin ok), full job lifecycle (submit→dispatch→DONE) on the
cluster. Also fixed stale build deps (libxce_deps missing librtti_dep,
scheduler_deps missing ssl_dep/c_capnproto_dep, unity include path).
Co-Authored-By: Claude <noreply@anthropic.com>
Deep-dive: the rejoin works but crashes ~3-8s later with a REAL heap corruption
(ASAN bad-free freeing a pointer 56B into a 60B region, all-loader frames;
non-ASAN debug build aborts with free(): invalid pointer, gdb stack corrupted).
The endpoint-unmonitor fix (correct, matches xce_ctx_stop) did not help.
Candidate roots: config re-push re-running sched_init_config non-idempotently
(sharedInst MOVE_PTR/zarray_put), lib reconnect vs in-flight vtask, or
state-resync merge. All scheduler reconnect changes reverted to keep the
scheduler stable; this is a dedicated deep-debugging task.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the PEND reason taxonomy (docs/pend-reason-investigation-design.md) to
left_scheduler_work 八.4 推迟/未决, with the three open decisions for unified
analysis: code strategy (reuse 3 codes + subcodes vs expand PREASON_* to ~20),
wiring priority (P0 main_reason + limit attach), and bjobs -p-equivalent MCP
capability.
Co-Authored-By: Claude <noreply@anthropic.com>
Full Agent chain verified on t1: mcp_operator API key → mcp-server
(scheduler_submit/get_job) → apiserver REST → controller → scheduler →
worker, job submit→run→DONE; cancel(kill)→EXIT. Note the apiserver
controller-restart rejoin gap (DataStore stays stale until apiserver
restart).
Co-Authored-By: Claude <noreply@anthropic.com>
bug-1510 (dispatch-fail re-PEND, 6928f74) + bug-1511 (sched_update_workers
in-place update on re-push, 1a2bd3f) together close bug-1494. Update
left_scheduler_work.md and buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
Document that the scheduler dispatch-fail re-PEND fix (commit 6928f74)
resolves one root cause of the "all jobs PEND" symptom; the
controller-restart self-heal-reconnect aspect of bug-1494 remains open.
Log bug-1510 in buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
Two stacked defects in the apiserver job-control link, both of which kept
hold/release/cancel from ever reaching the controller:
1. controllers/jobs.go: validateJobControl and foundJob asserted
jobinfo.Data["array_idx"].(int64), but StoreToOrderMaps produces
array_idx as int (models.JobInfo.ArrayIdx is declared int) -> panic
-> Gin Recovery -> HTTP 500 before backend.ControlJob ran. Add
normalizeJobInt64 / jobArrayIdx helpers and use them in both
functions (job_status already had this fix via normalizeJobStatus;
array_idx was missed).
2. backend/clib.go: backend.ControlJob set xce_jobctrl_t.action to the
models.JOB_* constant (JOB_STOP=1 / JOB_RESUME=2 / JOB_KILL=3 /
JOB_CHKPNT=8), but that field is the capnp JobControl union
discriminant (JobControl_u_siginfo=0 / JobControl_u_ckptoption=1).
The controller's job_ctrl() selected the wrong union branch (or
none, since encode_JobControl drops the union pointer for
non-0/1 discriminants) and rejected every control with -EINVAL.
Add jobCtrlDiscriminant() and use it.
Also: resolveExecUser was defined in the cgo-only convert.go but is
exercised by a non-cgo test, so `make test` (CGO_ENABLED=0) failed to
build the backend package. Move it to convert_pure.go so it compiles in
both modes.
Tests: 3 controller regression tests (validateJobControl allowed /
not-allowed branches, foundJob array-child branch) + 1 backend
jobCtrlDiscriminant test. make test fully green. Cluster E2E verified:
hold -> PSUSP, release -> PEND, cancel -> EXIT on a PEND job, no 500,
no -EINVAL, all daemons healthy.
Co-Authored-By: Claude <noreply@anthropic.com>
