myxce/.wolf/bugs/bug-1968.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1968	fixed	2026-10-10		apiserver	wrong-result	2026-10-10T11:55:48Z	2026-10-10T11:55:48Z	1	src/frontend/apiserver/controllers/jobs.go	apiserver restart submit	bug-1936	6a2c69f8	399	open	a735a6546a0b

bug-1968: apiserver job restart copies asked hosts, begin/term time and I/O paths it can never pass on

Description

Observed: (source reading, 2026-10-10; found while fixing bug-1936) RestartJobByID() (src/frontend/apiserver/controllers/jobs.go:832) builds the new job from the request plus the old job's row, and copies options that bug-1936 now refuses at submit because no field carries them to the controller or worker:

if newJob.AskedHosts == "" {
    if v, ok := oldJob["asked_hosts"].(string); ok { newJob.AskedHosts = v }
}
if v, ok := oldJob["begin_time"].(time.Time); ok { ... }   // also term_time,
if v, ok := oldJob["out_dir"].(string); ok { ... }         // in_file, err_file, out_file

preCheckJobRestartRequest() (:1350) validates -m asked hosts but does not refuse them the way preCheckSubmitOptions() (:1307, bug-1936) does for submit, so xcecli jobs restart -m host ... returns success and the host request is silently dropped.

Expected: restart, like submit, either passes each option on to the job or rejects it with 400 (preCheckSubmitOptions-style); options copied from the old job that cannot be honoured are not silently kept in apiserver memory only. Impact: wrong result: a restarted job ignores the asked hosts, begin/term time and I/O paths the user gave, with no error. Found by: review during the bug-1936 fix, 2026-10-10.

UPDATE (2026-10-10, during the fix): the same rebuild also lost options the job DOES get. Rows come from store.StructToOrderMap, which keeps each field's Go type, so oldJob["chkpnt_info"].(string) and oldJob["extend_data"].(string) never matched models.Jsonb: env/limits (exdata) were dropped on every restart and a checkpointed job never restarted from its checkpoint (restart_command, XCE_JOB_CHECKPOINT_PID). It also read oldJob["res_req"] (the column is resreq), dropping the resource requirement, and never copied exec_home.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (n/a: Go apiserver, CGO_ENABLED=0 tests)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ edad8c73-dirty
• Deployment: any (found by source reading)

Reproduction

1. Submit a job, let it finish.
2. POST /xce/v1/jobs/<jobid>/restart with body {"asked_hosts":"<host>"} (xcecli jobs restart has no -m flag and sends none), or restart a job whose row has out_file/out_dir set.
3. The restart returns success; the new job is placed without regard to <host> (scheduler never receives asked hosts).

• Frequency: deterministic (code path)
• Evidence: the code quoted in Description; no logs kept.

Root cause

RestartJobByID() (src/frontend/apiserver/controllers/jobs.go) rebuilds the new job by copying fields out of the old job's stored row. Its copy list predates bug-1936: it accepted asked_hosts in the request and copied asked hosts, begin/term time, in/out/err files and out_dir, none of which any job gets. The list was also never checked against the row's real keys and types: JSONB columns are models.Jsonb (not string) and the resource requirement key is resreq (not res_req), so those copies silently did nothing. Pre-existing, origin not bisected.

Fix

controllers/jobs.go:

• preCheckJobRestartRequest() refuses a non-blank asked_hosts with 400 ("job restart option asked_hosts not supported"), like preCheckSubmitOptions() (bug-1936); the request field stays so the client gets that message, not "unknown field". processJobRestartRequest() (which only copied asked hosts) is removed.
• The restart no longer carries over asked hosts, begin/term time, in/out/err files or out_dir.
• New rowJSON() reads a JSONB row column as models.Jsonb or string; used for chkpnt_info and extend_data, so a checkpointed job restarts with its restart command and checkpoint pid, and every restart keeps the old job's extended data (env, limits).
• Copies resreq (was res_req) and exec_home.

Verification

• Test: TestJobHandler_RestartRejectsAskedHosts, TestJobHandler_RestartCarriesOnlyHonouredOptions, TestJobHandler_RestartFromCheckpoint (src/frontend/apiserver/controllers/jobs_handler_test.go): RED before (jobs_handler_test.go:1537: status = 200, want 400; :1592: restarted job kept asked_hosts, which no job gets (also in_file/out_file/err_file/out_dir); :1597: ExtendData = "", want ...; :1606: ResReq = "", want rusage[mem=100]; :1637: Command = "sleep 1", want the checkpoint's restart command), GREEN after; with the fix stashed the three fail again. TestJobHandler_RestartNamedJob (bug-1898) still passes.
• Suites: xce-build.sh apiserver asan xce.build --incremental (CGO build; apiserver + xcecli Go tests all ok), 2026-10-10; apiserver CGO_ENABLED=0 go test ./... all packages ok
• Live: n/a: handler-level fixture tests only; no live restart of a checkpointed job was run (needs CRIU on a real worker)

References

• Fix commit: 6a2c69f8
• Related: bug-1936 (submit-side rejection), bug-1898 (restart identity)