Deep-dive: the rejoin works but crashes ~3-8s later with a REAL heap corruption
(ASAN bad-free freeing a pointer 56B into a 60B region, all-loader frames;
non-ASAN debug build aborts with free(): invalid pointer, gdb stack corrupted).
The endpoint-unmonitor fix (correct, matches xce_ctx_stop) did not help.
Candidate roots: config re-push re-running sched_init_config non-idempotently
(sharedInst MOVE_PTR/zarray_put), lib reconnect vs in-flight vtask, or
state-resync merge. All scheduler reconnect changes reverted to keep the
scheduler stable; this is a dedicated deep-debugging task.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the PEND reason taxonomy (docs/pend-reason-investigation-design.md) to
left_scheduler_work 八.4 推迟/未决, with the three open decisions for unified
analysis: code strategy (reuse 3 codes + subcodes vs expand PREASON_* to ~20),
wiring priority (P0 main_reason + limit attach), and bjobs -p-equivalent MCP
capability.
Co-Authored-By: Claude <noreply@anthropic.com>
Full Agent chain verified on t1: mcp_operator API key → mcp-server
(scheduler_submit/get_job) → apiserver REST → controller → scheduler →
worker, job submit→run→DONE; cancel(kill)→EXIT. Note the apiserver
controller-restart rejoin gap (DataStore stays stale until apiserver
restart).
Co-Authored-By: Claude <noreply@anthropic.com>
bug-1510 (dispatch-fail re-PEND, 6928f74) + bug-1511 (sched_update_workers
in-place update on re-push, 1a2bd3f) together close bug-1494. Update
left_scheduler_work.md and buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
Document that the scheduler dispatch-fail re-PEND fix (commit 6928f74)
resolves one root cause of the "all jobs PEND" symptom; the
controller-restart self-heal-reconnect aspect of bug-1494 remains open.
Log bug-1510 in buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
Two stacked defects in the apiserver job-control link, both of which kept
hold/release/cancel from ever reaching the controller:
1. controllers/jobs.go: validateJobControl and foundJob asserted
jobinfo.Data["array_idx"].(int64), but StoreToOrderMaps produces
array_idx as int (models.JobInfo.ArrayIdx is declared int) -> panic
-> Gin Recovery -> HTTP 500 before backend.ControlJob ran. Add
normalizeJobInt64 / jobArrayIdx helpers and use them in both
functions (job_status already had this fix via normalizeJobStatus;
array_idx was missed).
2. backend/clib.go: backend.ControlJob set xce_jobctrl_t.action to the
models.JOB_* constant (JOB_STOP=1 / JOB_RESUME=2 / JOB_KILL=3 /
JOB_CHKPNT=8), but that field is the capnp JobControl union
discriminant (JobControl_u_siginfo=0 / JobControl_u_ckptoption=1).
The controller's job_ctrl() selected the wrong union branch (or
none, since encode_JobControl drops the union pointer for
non-0/1 discriminants) and rejected every control with -EINVAL.
Add jobCtrlDiscriminant() and use it.
Also: resolveExecUser was defined in the cgo-only convert.go but is
exercised by a non-cgo test, so `make test` (CGO_ENABLED=0) failed to
build the backend package. Move it to convert_pure.go so it compiles in
both modes.
Tests: 3 controller regression tests (validateJobControl allowed /
not-allowed branches, foundJob array-child branch) + 1 backend
jobCtrlDiscriminant test. make test fully green. Cluster E2E verified:
hold -> PSUSP, release -> PEND, cancel -> EXIT on a PEND job, no 500,
no -EINVAL, all daemons healthy.
Co-Authored-By: Claude <noreply@anthropic.com>
