lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

attach jobs end with their command: relay watcher, once-only start banner, worker-side ttyd cleanup

Browse source 
Three attach-mode (-I) fixes, verified end-to-end on the t1 cluster:

1. Session hang on job finish (job_attach.go): the relay's client->worker
   pump blocked forever in ReadMessage after the worker hop closed, so the
   old wg.Wait()-before-Close teardown never ran and the client hung, then
   died with an abnormal 1006. A broker watcher now subscribes to
   EventJobFinished and on job end sends the 4001 policy close and closes
   both hops, unblocking the pumps; the teardown follows normally. The
   client (attach.go) treats 4001/1000/1001 -- and 1006-after-output as a
   final safety net -- as a clean session end.

2. "<<Starting on t1 >>" printed repeatedly (submit.go): the status poll
   printed the banner on every transition; now gated on a reportedStart
   flag. The URL is also no longer printed in attach mode -- it only makes
   sense for browse (-I --tty --browse) and DISPLAY jobs.

3. ttyd left behind after an attach job exits (xcecd_worker.c): a new
   tty_stop_attach_session(), called from process_exit_callback for TTY
   jobs, kills the per-job ttyd right away. This is insurance -- the sh -c
   exec means uv_proc monitors ttyd's own pid, so ttyd is normally gone
   before the callback fires -- but it guarantees no orphan when the
   timing lines up badly. Web-viewed sessions are untouched: they keep
   ttyd and output alive for later browsing, per design.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-15 08:32:21 +00:00
parent 601f04dc00
commit 66a1d1e680
4 changed files with 146 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

39
src/worker/src/xcecd_worker.c
View file

@ -60,6 +60,7 @@ static void check_and_reconnect();
static void worker_reconnect_callback(bool connected);
static int xjob_start_vnc();
static void xjob_stop_vnc();
static void tty_stop_attach_session();
static bool check_process_status(pid_t target_pid, int expected_signal,
int timeout_seconds);
@ -515,6 +516,12 @@ static void process_exit_callback(int pid, int exit_status, int term_signal,
xjob_stop_vnc();
}
/* attach-mode TTY job: the session ends with the command (web-viewed
* TTY jobs keep ttyd for later browsing -- see tty_stop_attach_session) */
if (TTY == mydata.transfer_info->interactive_mode) {
tty_stop_attach_session();
}
if (job_runtime_info.job_post_exec[0] != '\0') {
job_post_exec();
}
@ -710,6 +717,38 @@ static void xjob_stop_vnc()
jobidStr);
}
/* An attached TTY job (interactive=1, the xcecli attach path) is done the
* moment its command exits: the user watches the same terminal, so there is
* nothing left to serve. Kill the ttyd we spawned for the job right away --
* ttyd's own -Q teardown cannot be relied on here: with its uv/libwebsockets
* foreign event loop it logs "exiting due to --exit-with-program" but stays
* alive while a websocket client (the apiserver relay) is still connected,
* which would leave the attach hanging and the ttyd orphaned.
*
* Web-viewed sessions (--browse / DISPLAY) intentionally keep ttyd and the
* output alive after the command ends until a browser connects or the job
* is reaped; tty_stop_attach_session is only called for the attach path. */
static void tty_stop_attach_session()
{
pid_t ttyd_pid = uv_proc_get_pid(&uv_proc_manager);
if (ttyd_pid <= 0) {
return;
}
char jobidStr[64];
fmt_jobid(jobidStr, sizeof(jobidStr), jobid);
XCECD_WORKER_LOG(XCE_LOG_DEBUG, "Terminating ttyd %d of attach job %s",
ttyd_pid, jobidStr);
signal_children_recursive(SIGKILL, ttyd_pid);
if (kill(ttyd_pid, SIGKILL) != 0 && errno != ESRCH) {
XCECD_WORKER_LOG(XCE_LOG_WARN,
"Failed to kill ttyd %d of job %s: %s", ttyd_pid,
jobidStr, strerror(errno));
}
}
static void worker_job_callback(job_assignment_t* info)
{
char* job_cmd = NULL;