docs: document graceful job kill/stop and job.terminate_interval (bug-1957)

- src/docs/xce.md: explicit signal and force kill are REST-only (signal,
  force_kill); xcecli has no signal option yet.
- src/docs/deployment.md: app.yaml row lists the job block
  (max_array_size, terminate_interval) and trusted proxies.
- xcecli jobs kill/stop/resume help describes the signal sequences.
- REST: control endpoint annotations and the signal/force_kill model fields
  describe the behaviour (swagger output not regenerated: swag fails on an
  unrelated annotation).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

src/docs/deployment.md

@@ -234,7 +234,7 @@ apiserver -d /opt/xce/apiserver
 
 | File | Purpose |
 |---|---|
-| `app.yaml` | HTTP(S) port, JWT secret/expiry, rate limits, CORS, IP whitelist, and the `backend:` block (`address`, `port`, `from`, `key`, optional `peers` for HA) pointing at the controller. |
+| `app.yaml` | HTTP(S) port, JWT secret/expiry, rate limits, CORS, IP whitelist, trusted proxies, the `job:` block (`max_array_size`; `terminate_interval`, the seconds between the signals of a plain kill or stop, default 10, see `xce.md` "Killing and stopping jobs"), and the `backend:` block (`address`, `port`, `from`, `key`, optional `peers` for HA) pointing at the controller. |
 | `auth.yaml` | Auth provider selection (`config_file` \| `pam` \| `ldap` \| `oauth2`); for `config_file`, embeds usernames/passwords/roles directly. |
 | `rbac.yaml` | Permission catalog + role→permission mappings (domains: `self`, `self_ug`, etc.). User→role binding comes from `auth.yaml`, not this file. |
 | `logger.yaml` | zap logger level/encoding/rotation. |

src/docs/xce.md

@@ -64,8 +64,9 @@ Job control follows LSF's TERMINATE and SUSPEND actions:
 - **Stop** (`xcecli jobs stop <id>`): SIGTSTP first, so the job can catch
   it, prepare and stop itself; SIGSTOP follows after the interval if it has
   not stopped. **Resume** sends SIGCONT and cancels a stop still in progress.
-- **Explicit signal** (kill with a signal): that signal only, once.
-- **Force kill**: SIGKILL at once.
+- **Explicit signal**: that signal only, once (REST API: `signal` in the
+  control request, kill action; `xcecli` has no signal option yet).
+- **Force kill**: SIGKILL at once (REST API: `force_kill: true`).
 
 The interval is the API server's `job.terminate_interval` setting
 (`config/app.yaml`, seconds; default 10 when unset or 0, as LSF's

src/frontend/apiserver/controllers/jobs.go

@@ -2036,7 +2036,11 @@ func postJobRequest(job *models.JobInfo) error {
 
 // ControlJobByID godoc
 // @Summary Perform an action on a job
-// @Description Submit an action to be performed on a specific job identified by jobids
+// @Description Submit an action to be performed on a specific job identified by jobids.
+// @Description kill without a signal: SIGINT, SIGTERM, then SIGKILL, job.terminate_interval
+// @Description seconds apart while the job runs; with a signal: that signal once; with
+// @Description force_kill: SIGKILL at once. stop: SIGTSTP, then SIGSTOP after the interval
+// @Description if the job has not stopped. resume: SIGCONT (cancels a stop in progress).
 // @Tags jobs
 // @Accept json
 // @Produce json
@@ -2313,6 +2317,7 @@ func (j JobController) ControlJobByID(ctx *gin.Context) {
 // ControlJob godoc
 // @Summary Perform an action on filtered out jobs
 // @Description Submit an action to be performed on  specific jobs identified by filter in request body.
+// @Description kill/stop/resume signals as for POST /jobs/{jobids}/control.
 // @Tags jobs
 // @Accept json
 // @Produce json

src/frontend/apiserver/models/jobs.go

@@ -761,18 +761,18 @@ type JobControlFilter struct {
 type JobControlRequest struct {
 	Filter    JobControlFilter `json:"filter" binding:"required"`
 	Action    string           `json:"action" binding:"required"`
-	Signal    string           `json:"signal,omitempty"`
+	Signal    string           `json:"signal,omitempty"` // kill only: send this signal once instead of the graceful sequence
 	Comment   string           `json:"comment,omitempty"`
-	ForceKill bool             `json:"force_kill,omitempty"`
-	Method    string           `json:"method,omitempty"` // "criu" as default
+	ForceKill bool             `json:"force_kill,omitempty"` // kill: SIGKILL at once (no graceful sequence)
+	Method    string           `json:"method,omitempty"`     // "criu" as default
 }
 
 type JobControlJobIDRequest struct {
 	Action    string `json:"action" binding:"required"`
-	Signal    string `json:"signal,omitempty"` // signal value
+	Signal    string `json:"signal,omitempty"` // kill only: send this signal once instead of the graceful sequence
 	Comment   string `json:"comment,omitempty"`
-	ForceKill bool   `json:"force_kill,omitempty"`
-	Method    string `json:"method,omitempty"` // "criu" as default
+	ForceKill bool   `json:"force_kill,omitempty"` // kill: SIGKILL at once (no graceful sequence)
+	Method    string `json:"method,omitempty"`     // "criu" as default
 }
 
 type JobControlEvent struct {

src/frontend/xcecli/cmd/jobs/controller.go

@@ -151,6 +151,11 @@ func NewKillJobsCmd(configManager *config.ConfigManager) *cobra.Command {
 		Use:   "kill",
 		Short: "Kill jobs",
 		Long: `Kill jobs by job IDs or filters.
+
+The job gets SIGINT, then SIGTERM, then SIGKILL, one interval apart (the
+API server's job.terminate_interval, default 10 s), each only while it is
+still running, so it can catch SIGINT/SIGTERM and clean up.
+
 Examples:
   xce jobs kill jobid1,jobid2,...      # kill jobs by job IDs
   xce jobs kill <array-id>[k]          # kill element k of an array
@@ -190,6 +195,11 @@ func NewStopJobsCmd(configManager *config.ConfigManager) *cobra.Command {
 		Use:   "stop",
 		Short: "Stop jobs",
 		Long: `Stop jobs by job IDs or filters.
+
+The job gets SIGTSTP first, so it can catch it, prepare and stop itself;
+SIGSTOP follows after the interval (job.terminate_interval, default 10 s)
+if it has not stopped.
+
 Examples:
   xce jobs stop jobid1,jobid2,...      # stop jobs by job IDs
   xce jobs stop <array-id>[k]          # stop element k of an array
@@ -229,6 +239,9 @@ func NewResumeJobsCmd(configManager *config.ConfigManager) *cobra.Command {
 		Use:   "resume",
 		Short: "Resume jobs",
 		Long: `Resume jobs by job IDs or filters.
+
+The job gets SIGCONT; a stop still in progress is cancelled.
+
 Examples:
   xce jobs resume jobid1,jobid2,...      # resume jobs by job IDs
   xce jobs resume <array-id>[k]          # resume element k of an array