lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

worker: reap unattached TTY jobs via ttyd watchdog with real-pid resolution

Browse source 
An attach job (xcecli -I without --browse) that no client ever connects
to sat in RUN forever: ttyd only starts the job command on the first
websocket client, so with no client the command never runs, ttyd never
exits, and the job holds its ttyd, port and per-job worker indefinitely.

New exdata key "attach":1, set by the CLI for TTY jobs that are not
--browse and carried through the apiserver's extendDataMap to the
worker's transfer_info->attach_mode. For such jobs the worker starts a
1 s watchdog when the job process starts:

- no client attached within 60 s -> reap ttyd (SIGKILL to the monitored
  tree), job EXITs with status message "No client attached within 60 s"
- client came and the command ran and exited -> give ttyd 5 s to stop
  on its own (-Q), then reap: "Command exited; ttyd did not stop on its
  own"

The reap lands in the regular process_exit_callback path (JOB_EXITED,
status reason, post-exec, worker exit), so kill and cleanup behave like
any other job termination.

Field-critical detail: uv_proc_get_pid returns the "/bin/sh -c ttyd ..."
wrapper pid, and sh does not reliably exec -- on the field ttyd runs as
a child of the wrapper. Judging the wrapper's children would see ttyd
itself as "the command has started" and never reap. The watchdog
resolves the real ttyd pid first (/proc/<pid>/comm walk down through
the sh wrapper, capped depth) and judges client activity by the real
ttyd's children; killing still goes through the monitored wrapper tree
so both topologies (exec'd or forked) are covered.

Web-viewed sessions (--browse / DISPLAY) are untouched: no attach
exdata, no watchdog -- ttyd stays until a browser connects or the job
is killed, as before.

Attach failure on the CLI side prints that the job will be cleaned up
by the worker if no client attaches within 60 s.

Tests: xcecli (attach flag matrix), apiserver controllers (attach
exdata end-to-end in ExtendData), worker native suites; live t1
cluster: normal attach completes DONE with no residue, unattached job
EXITs at ~60 s with the reason in the worker log and port released,
Ctrl-C-then-wait completes when the command ends, --browse job stays
RUN past 60 s and kills cleanly.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-16 03:38:24 +00:00
parent 6ce7bdbe11
commit 4f188310ba
9 changed files with 421 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

8
src/frontend/apiserver/controllers/jobs.go
View file

@ -2048,6 +2048,14 @@ func processJobRequest(ctx *gin.Context, req *models.JobSubmitRequest, defaultVa
extendDataMap["interactive"] = req.Interactive
}
// attach=1: a TTY job whose session ends with its command (xcecli -I
// without --browse). The worker uses it to reap ttyd when no client ever
// connects -- without it an unattached job would sit in RUN forever,
// holding its ttyd, port and per-job worker.
if req.Attach != 0 {
extendDataMap["attach"] = req.Attach
}
if len(extendDataMap) > 0 {
updatedExtendDataBytes, err := json.Marshal(extendDataMap)
if err != nil {

50
src/frontend/apiserver/controllers/jobs_handler_test.go
View file

@ -428,6 +428,56 @@ func TestJobHandler_SmokeSubmitErrorIs500(t *testing.T) {
}
}
// TestJobHandler_SubmitInteractiveAttachExdata checks the submit handler
// packs interactive/attach into the job's ExtendData: attach=1 only for the
// terminal-attach TTY kind (the worker reaps ttyd when no client ever
// connects), never for --browse or other interactive modes.
func TestJobHandler_SubmitInteractiveAttachExdata(t *testing.T) {
cases := []struct {
name string
req map[string]interface{}
wantAttach bool
}{
{"tty attach", map[string]interface{}{
"job_name": "a", "command": "sleep 1", "interactive": 1, "attach": 1,
}, true},
{"tty web view (no attach)", map[string]interface{}{
"job_name": "a", "command": "sleep 1", "interactive": 1,
}, false},
{"xjob with attach ignored worker-side but passed", map[string]interface{}{
"job_name": "a", "command": "sleep 1", "interactive": 2, "attach": 1,
}, true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
f := newHandlerFixture(t)
f.seedQueue("normal")
w := f.submit(tc.req)
if w.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body: %s", w.Code, w.Body.String())
}
subs := f.submissions()
if len(subs) != 1 {
t.Fatalf("submitJob calls = %d, want 1", len(subs))
}
var ex map[string]interface{}
if raw := subs[0].Job.ExtendData.String; raw != "" {
if err := json.Unmarshal([]byte(raw), &ex); err != nil {
t.Fatalf("ExtendData %q is not JSON: %v", raw, err)
}
}
if got, ok := ex["interactive"]; !ok || int(got.(float64)) != tc.req["interactive"].(int) {
t.Errorf("ExtendData interactive = %v (%v), want %d", got, ok, tc.req["interactive"])
}
_, hasAttach := ex["attach"]
if hasAttach != tc.wantAttach {
t.Errorf("ExtendData attach present = %v, want %v (exdata: %s)", hasAttach, tc.wantAttach, subs[0].Job.ExtendData.String)
}
})
}
}
// TestJobHandler_SmokeControlByID controls one seeded ordinary job by ID
// and checks the dispatch is recorded once with its unique_job_id.
func TestJobHandler_SmokeControlByID(t *testing.T) {

5
src/frontend/apiserver/models/jobs.go
View file

@ -669,6 +669,11 @@ type JobSubmitRequest struct {
Exclusive bool `json:"exclusive,omitempty"`
Rerunable bool `json:"rerunable,omitempty"`
Interactive int `json:"interactive,omitempty"`
// Attach marks an interactive TTY job as the terminal-attach kind
// (xcecli -I without --browse): the session ends with the command, so
// the worker may reap the ttyd when no client ever connects. 0 keeps
// the web-viewed semantics.
Attach int `json:"attach,omitempty"`
WarningTimePeriod int `json:"warning_time_period,omitempty"`
WarningAction string `json:"warning_action,omitempty"`
DataSpecFile string `json:"data_spec_file,omitempty"`

12
src/frontend/xcecli/cmd/jobs/submit.go
View file

@ -253,6 +253,12 @@ func (js *JobSubmitter) BuildJobRequest() (*models.JobSubmitRequest, error) {
if js.interactive {
if js.useTTY {
jobReq.Interactive = 1 // TTY mode
// attach jobs end with their command, web-viewed ones keep the
// session alive for a later browser; the worker needs the
// distinction to reap a ttyd no client ever connected to
if !js.useBrowse {
jobReq.Attach = 1
}
} else if js.useXJob {
jobReq.Interactive = 2 // XJob mode
} else {
@ -520,7 +526,11 @@ func (js *JobSubmitter) RunJobsSubmit() error {
// user asked for the web view. --browse opens the ttyd web URL
// in the browser instead of attaching.
if !js.useBrowse {
return runAttachSession(serverInfo.URL, serverInfo.Token, jobID)
err := runAttachSession(serverInfo.URL, serverInfo.Token, jobID)
if err != nil {
fmt.Println("note: the job will be cleaned up by the worker if no client attaches within 60 s")
}
return err
}
if err := openBrowser(url); err != nil {
errMsg := request.ExtractErrorMessage(err)

49
src/frontend/xcecli/cmd/jobs/submit_test.go
View file

@ -133,6 +133,47 @@ func TestSubmitHelpShowsArrayForms(t *testing.T) {
}
}
// Attach marks the terminal-attach kind of TTY job so the worker may reap
// ttyd when no client ever connects; --browse keeps the web-viewed
// semantics and must not set it.
func TestBuildJobRequest_AttachFlagMatrix(t *testing.T) {
cases := []struct {
name string
interactive bool
useTTY bool
useBrowse bool
useXJob bool
wantInter int
wantAttach int
}{
{"batch job", false, false, false, false, 0, 0},
{"tty attach (bare -I)", true, true, false, false, 1, 1},
{"tty browse", true, true, true, false, 1, 0},
{"xjob", true, false, false, true, 2, 0},
{"display (interactive=3)", true, false, false, false, 3, 0},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
js := NewJobSubmitter(nil)
js.command = "sleep 1"
js.interactive = tc.interactive
js.useTTY = tc.useTTY
js.useBrowse = tc.useBrowse
js.useXJob = tc.useXJob
req, err := js.BuildJobRequest()
if err != nil {
t.Fatalf("BuildJobRequest: %v", err)
}
if req.Interactive != tc.wantInter {
t.Errorf("Interactive = %d, want %d", req.Interactive, tc.wantInter)
}
if req.Attach != tc.wantAttach {
t.Errorf("Attach = %d, want %d", req.Attach, tc.wantAttach)
}
})
}
}
// jobStateServer answers the per-job status query with the states fed in
// sequence, holding the last one forever; every request's arrival time is
// recorded so tests can assert the poll cadence.
@ -167,9 +208,9 @@ func newJobStateServer(t *testing.T, states ...string) *jobStateServer {
_ = json.NewEncoder(w).Encode(models.JobsResponse{
Code: 0,
Data: struct {
Count int `json:"count"`
Count int `json:"count"`
Rows []models.QueryJob `json:"rows"`
Page_token string `json:"page_token"`
Page_token string `json:"page_token"`
}{Rows: []models.QueryJob{{JobStatus: held, InteractiveJobURL: "http://h:9000"}}},
})
}))
@ -250,9 +291,9 @@ func TestWaitForInteractiveURL_RunRaceUsesFastInterval(t *testing.T) {
_ = json.NewEncoder(w).Encode(models.JobsResponse{
Code: 0,
Data: struct {
Count int `json:"count"`
Count int `json:"count"`
Rows []models.QueryJob `json:"rows"`
Page_token string `json:"page_token"`
Page_token string `json:"page_token"`
}{Rows: []models.QueryJob{row}},
})
}))

5
src/frontend/xcecli/models/jobs.go
View file

@ -343,6 +343,11 @@ type JobSubmitRequest struct {
Exclusive bool `json:"exclusive,omitempty"`
Rerunable bool `json:"rerunable,omitempty"`
Interactive int `json:"interactive,omitempty"`
// Attach marks an interactive TTY job as the terminal-attach kind (xcecli
// -I without --browse): its session ends with the command, so the worker
// may reap the ttyd when no client ever connects. 0 (default) keeps the
// web-viewed semantics.
Attach int `json:"attach,omitempty"`
WarningTimePeriod int `json:"warning_time_period,omitempty"`
WarningAction string `json:"warning_action,omitempty"`
DataSpecFile string `json:"data_spec_file,omitempty"`