worker: reap unattached TTY jobs via ttyd watchdog with real-pid resolution

An attach job (xcecli -I without --browse) that no client ever connects
to sat in RUN forever: ttyd only starts the job command on the first
websocket client, so with no client the command never runs, ttyd never
exits, and the job holds its ttyd, port and per-job worker indefinitely.

New exdata key "attach":1, set by the CLI for TTY jobs that are not
--browse and carried through the apiserver's extendDataMap to the
worker's transfer_info->attach_mode. For such jobs the worker starts a
1 s watchdog when the job process starts:

- no client attached within 60 s -> reap ttyd (SIGKILL to the monitored
  tree), job EXITs with status message "No client attached within 60 s"
- client came and the command ran and exited -> give ttyd 5 s to stop
  on its own (-Q), then reap: "Command exited; ttyd did not stop on its
  own"

The reap lands in the regular process_exit_callback path (JOB_EXITED,
status reason, post-exec, worker exit), so kill and cleanup behave like
any other job termination.

Field-critical detail: uv_proc_get_pid returns the "/bin/sh -c ttyd ..."
wrapper pid, and sh does not reliably exec -- on the field ttyd runs as
a child of the wrapper. Judging the wrapper's children would see ttyd
itself as "the command has started" and never reap. The watchdog
resolves the real ttyd pid first (/proc/<pid>/comm walk down through
the sh wrapper, capped depth) and judges client activity by the real
ttyd's children; killing still goes through the monitored wrapper tree
so both topologies (exec'd or forked) are covered.

Web-viewed sessions (--browse / DISPLAY) are untouched: no attach
exdata, no watchdog -- ttyd stays until a browser connects or the job
is killed, as before.

Attach failure on the CLI side prints that the job will be cleaned up
by the worker if no client attaches within 60 s.

Tests: xcecli (attach flag matrix), apiserver controllers (attach
exdata end-to-end in ExtendData), worker native suites; live t1
cluster: normal attach completes DONE with no residue, unattached job
EXITs at ~60 s with the reason in the worker log and port released,
Ctrl-C-then-wait completes when the command ends, --browse job stays
RUN past 60 s and kills cleanly.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-16 03:38:24 +00:00
parent 6ce7bdbe11
commit 4f188310ba
9 changed files with 421 additions and 12 deletions

View file

@ -133,6 +133,47 @@ func TestSubmitHelpShowsArrayForms(t *testing.T) {
}
}
// Attach marks the terminal-attach kind of TTY job so the worker may reap
// ttyd when no client ever connects; --browse keeps the web-viewed
// semantics and must not set it.
func TestBuildJobRequest_AttachFlagMatrix(t *testing.T) {
cases := []struct {
name string
interactive bool
useTTY bool
useBrowse bool
useXJob bool
wantInter int
wantAttach int
}{
{"batch job", false, false, false, false, 0, 0},
{"tty attach (bare -I)", true, true, false, false, 1, 1},
{"tty browse", true, true, true, false, 1, 0},
{"xjob", true, false, false, true, 2, 0},
{"display (interactive=3)", true, false, false, false, 3, 0},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
js := NewJobSubmitter(nil)
js.command = "sleep 1"
js.interactive = tc.interactive
js.useTTY = tc.useTTY
js.useBrowse = tc.useBrowse
js.useXJob = tc.useXJob
req, err := js.BuildJobRequest()
if err != nil {
t.Fatalf("BuildJobRequest: %v", err)
}
if req.Interactive != tc.wantInter {
t.Errorf("Interactive = %d, want %d", req.Interactive, tc.wantInter)
}
if req.Attach != tc.wantAttach {
t.Errorf("Attach = %d, want %d", req.Attach, tc.wantAttach)
}
})
}
}
// jobStateServer answers the per-job status query with the states fed in
// sequence, holding the last one forever; every request's arrival time is
// recorded so tests can assert the poll cadence.
@ -167,9 +208,9 @@ func newJobStateServer(t *testing.T, states ...string) *jobStateServer {
_ = json.NewEncoder(w).Encode(models.JobsResponse{
Code: 0,
Data: struct {
Count int `json:"count"`
Count int `json:"count"`
Rows []models.QueryJob `json:"rows"`
Page_token string `json:"page_token"`
Page_token string `json:"page_token"`
}{Rows: []models.QueryJob{{JobStatus: held, InteractiveJobURL: "http://h:9000"}}},
})
}))
@ -250,9 +291,9 @@ func TestWaitForInteractiveURL_RunRaceUsesFastInterval(t *testing.T) {
_ = json.NewEncoder(w).Encode(models.JobsResponse{
Code: 0,
Data: struct {
Count int `json:"count"`
Count int `json:"count"`
Rows []models.QueryJob `json:"rows"`
Page_token string `json:"page_token"`
Page_token string `json:"page_token"`
}{Rows: []models.QueryJob{row}},
})
}))