The apiserver stayed circuit-breaker-degraded after a full controller
restart until manually restarted. Two root causes:
1. Breaker recovery was decoupled from the breaker itself. The breaker can
be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while
the controller is electing) with no heartbeat ever failing, but
StartHeartbeat only recovered on a fail→success heartbeat transition
(prevFailed), so the breaker stayed degraded forever.
→ extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when
prevFailed OR the breaker is currently tripped. 5 new unit tests.
2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's
internal ctx state is XCE_CTX_ERR. After a controller restart whose
election outlasts the FAILOVER_CALL retry budget, the retry loop's
xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the
FF path never reconnects and the client stays disconnected forever.
→ lib: new xcecli_force_reconnect() forces a rotate+recover regardless
of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while
degraded.
Verified on cluster: after controller kill+restart, the apiserver logs
"backend re-joined controller after reconnect" and job submissions resume
without a manual restart; a fresh job runs to DONE. Controller, scheduler
(rejoin ok, bug-1513) and workers all auto-recover.
Co-Authored-By: Claude <noreply@anthropic.com>
The "libxce reconnect heap corruption" (ASAN bad-free, free(): invalid
pointer) was NOT a libxce bug — xce_ctx_reconnect is fine. The crashes are
in the scheduler's rejoin state re-sync path (config/dataset re-push),
fixed with the reconnect/rejoin feature now integrated:
1. group.c makeGroup refresh: FREEUP(m->u.name) raw-freed shared interned
xstrings (makeString is a global refcounted table); the correct release
is dereference(), matching group_killer. → dereference().
2. limit.c limit_update_config: after key_add returns -EEXIST (rejoin
re-push — limit name already present) it called kill_limit(limit), but
xkvmgr_add's vclear already freed the limit via the kvmgr datakiller
(=kill_limit). → skip on -EEXIST (idempotent re-push).
3. handler.c/scheduler.c: the config push was applied inline on the vcpu
thread (sched_init_config adds queues/plugin queues) while job updates
flow through pend_updates on the main loop — concurrent rejoin re-push
let fcfs_update_job see a NULL j->queue (SIGSEGV). Route the config
push through pend_updates (XCE_SCHED_OP_UPDATE_CONFIG, killer case
already existed) and process config requests FIRST in sched_update via
a staging queue so jobs always see their queues/resources.
Verified: 295 lib unit tests pass under ASAN (incl. 2 new group-refresh
regression tests), scheduler survives 3 consecutive controller restarts
(each rejoin ok), full job lifecycle (submit→dispatch→DONE) on the
cluster. Also fixed stale build deps (libxce_deps missing librtti_dep,
scheduler_deps missing ssl_dep/c_capnproto_dep, unity include path).
Co-Authored-By: Claude <noreply@anthropic.com>
Deep-dive: the rejoin works but crashes ~3-8s later with a REAL heap corruption
(ASAN bad-free freeing a pointer 56B into a 60B region, all-loader frames;
non-ASAN debug build aborts with free(): invalid pointer, gdb stack corrupted).
The endpoint-unmonitor fix (correct, matches xce_ctx_stop) did not help.
Candidate roots: config re-push re-running sched_init_config non-idempotently
(sharedInst MOVE_PTR/zarray_put), lib reconnect vs in-flight vtask, or
state-resync merge. All scheduler reconnect changes reverted to keep the
scheduler stable; this is a dedicated deep-debugging task.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the PEND reason taxonomy (docs/pend-reason-investigation-design.md) to
left_scheduler_work 八.4 推迟/未决, with the three open decisions for unified
analysis: code strategy (reuse 3 codes + subcodes vs expand PREASON_* to ~20),
wiring priority (P0 main_reason + limit attach), and bjobs -p-equivalent MCP
capability.
Co-Authored-By: Claude <noreply@anthropic.com>
Full Agent chain verified on t1: mcp_operator API key → mcp-server
(scheduler_submit/get_job) → apiserver REST → controller → scheduler →
worker, job submit→run→DONE; cancel(kill)→EXIT. Note the apiserver
controller-restart rejoin gap (DataStore stays stale until apiserver
restart).
Co-Authored-By: Claude <noreply@anthropic.com>
bug-1510 (dispatch-fail re-PEND, 6928f74) + bug-1511 (sched_update_workers
in-place update on re-push, 1a2bd3f) together close bug-1494. Update
left_scheduler_work.md and buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
Document that the scheduler dispatch-fail re-PEND fix (commit 6928f74)
resolves one root cause of the "all jobs PEND" symptom; the
controller-restart self-heal-reconnect aspect of bug-1494 remains open.
Log bug-1510 in buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
Two stacked defects in the apiserver job-control link, both of which kept
hold/release/cancel from ever reaching the controller:
1. controllers/jobs.go: validateJobControl and foundJob asserted
jobinfo.Data["array_idx"].(int64), but StoreToOrderMaps produces
array_idx as int (models.JobInfo.ArrayIdx is declared int) -> panic
-> Gin Recovery -> HTTP 500 before backend.ControlJob ran. Add
normalizeJobInt64 / jobArrayIdx helpers and use them in both
functions (job_status already had this fix via normalizeJobStatus;
array_idx was missed).
2. backend/clib.go: backend.ControlJob set xce_jobctrl_t.action to the
models.JOB_* constant (JOB_STOP=1 / JOB_RESUME=2 / JOB_KILL=3 /
JOB_CHKPNT=8), but that field is the capnp JobControl union
discriminant (JobControl_u_siginfo=0 / JobControl_u_ckptoption=1).
The controller's job_ctrl() selected the wrong union branch (or
none, since encode_JobControl drops the union pointer for
non-0/1 discriminants) and rejected every control with -EINVAL.
Add jobCtrlDiscriminant() and use it.
Also: resolveExecUser was defined in the cgo-only convert.go but is
exercised by a non-cgo test, so `make test` (CGO_ENABLED=0) failed to
build the backend package. Move it to convert_pure.go so it compiles in
both modes.
Tests: 3 controller regression tests (validateJobControl allowed /
not-allowed branches, foundJob array-child branch) + 1 backend
jobCtrlDiscriminant test. make test fully green. Cluster E2E verified:
hold -> PSUSP, release -> PEND, cancel -> EXIT on a PEND job, no 500,
no -EINVAL, all daemons healthy.
Co-Authored-By: Claude <noreply@anthropic.com>
