docs(spec): application-profile v1 attributes: job starter added, others ship with their features
- v1 profile attributes: mode (direct), launch type, env + pre/post-exec, essential resreq and limits, plus the job starter - principle: a profile attribute is added by the spec that builds its job feature; requeue/success exit values -> job-requeue (new backlog item), checkpoint and warning-before-kill later, output path -> I/O track, runtime estimate is not a profile attribute - log bug-1970: REST submit accepts rerunable, requeue_error_values, run_time_estimation, mig_threshold, chkpnt_*, warning_* and drops them Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
83c6eb8024
commit
67002b2a25
GPG key ID:
80533536B0EB651C
4 changed files with 134 additions and 4 deletions
|
|
@ -167,8 +167,26 @@ fits XCE's LSF-style job model.
|
|||
order), its task range (min / max), and per-job execution limits as
|
||||
maximums (memory, runtime, CPU, tasks), mapped onto the existing worker
|
||||
cgroup limits.
|
||||
5. **Job starter** (decided 2026-10-12): a wrapper the worker puts in
|
||||
front of the job command (e.g. `module load openfoam &&`,
|
||||
`apptainer exec img`), frozen on the dispatch record. Design decides
|
||||
whether it wraps the MPI launcher or each task.
|
||||
All of these are essentials under the additive rule; none is a
|
||||
convenience default.
|
||||
- **A profile attribute ships with its job feature** (user, 2026-10-12): a
|
||||
profile attribute is added by the spec that builds the job-level feature
|
||||
it configures, never ahead of it, so no profile setting is dead config.
|
||||
None of the other candidates exists as a job feature today, so they are
|
||||
deferred:
|
||||
- requeue exit values and success exit values: `job-requeue` (backlog);
|
||||
- checkpoint method/period/directory, and a warning signal before the run
|
||||
limit: their own future specs;
|
||||
- output path template: the PR-6/PR-7 I/O track (output options are
|
||||
rejected at submit today);
|
||||
- runtime estimate: not a profile attribute at all (it describes one run:
|
||||
a job option or preset; the profile's runtime limit is the maximum).
|
||||
- Found on the way: the REST submit accepts several of these options and
|
||||
silently drops them (bug-1970).
|
||||
- **Users pick a profile at submit** (REST, `xcecli jobs submit --app <name>`,
|
||||
harness, libxce). A **cluster default profile** applies when none is given.
|
||||
- **Jobs add, never override.** A job may add to its profile's values (more
|
||||
|
|
@ -552,9 +570,6 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
|
|||
(shared with `job-config-merge`).
|
||||
- Tests.
|
||||
- **Open for requirements**:
|
||||
- Which extra attributes go into v1 beyond the four ("maybe others"). Examples:
|
||||
requeue exit values, runtime estimate, job starter or wrapper, output path
|
||||
template, checkpoint settings.
|
||||
- ~~How resource defaults combine with a job's own resreq~~ — settled
|
||||
2026-10-09, owned by `job-config-merge`.
|
||||
- Profile versioning and audit.
|
||||
|
|
@ -569,7 +584,8 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
|
|||
pending jobs deferred to `job-modify`, profile aliases in v1; override granularity (no markers; additive rule),
|
||||
limits a job may lower (profile limits are maximums), personal presets
|
||||
(client side), `app:override` and inheritance (deferred); execution modes
|
||||
split into `job-sandbox-execution`.
|
||||
split into `job-sandbox-execution`; v1 attributes are the four plus the
|
||||
job starter, others ship with their job features.
|
||||
- **Out**:
|
||||
- The `sandbox` mode and its options, I/O directories, capability
|
||||
advertisement, placement by mode and the worker sandbox mechanics
|
||||
|
|
|
|||
|
|
@ -123,6 +123,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
|
|||
- [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
|
||||
- [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
|
||||
- [ ] job-modify — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Modify a pending job (LSF `bmod`): apiserver action (today `"modify"` is commented out, `controllers/jobs.go:2446`; `JOB_MODIFY` only an enum), controller opcode, WAL fact with snapshot field-completeness, scheduler re-merge, xcecli. First attributes: `--app` (owner with `app:use` on the new profile; admins in bulk, e.g. `--app-from <old>`), `-R`, `-n`, queue; later begin/end time (`job-time-window`), dependencies. Dependencies: application-profile, job-config-merge.
|
||||
- [ ] job-requeue — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Automatic requeue on given exit values (LSF `REQUEUE_EXIT_VALUES`) and success exit values (`SUCCESS_EXIT_VALUES`), with a requeue loop limit; wires the commented-out `requeue` job action (`apiserver/controllers/jobs.go`) and the REST `requeue_error_values` / `rerunable` options that are silently dropped today (bug-1970). Adds the matching application-profile attributes (a profile attribute ships with its job feature). Later, same principle: checkpoint settings and warning-before-kill each get their own spec. Dependencies: application-profile.
|
||||
- [ ] xcecli-presets — NEW 2026-10-12 (from application-profile discussion, user decisions). Client-side presets replace convenience defaults in profiles: a preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env, limits) expanded by the client, a flag replacing the preset's value for that option; one file per preset (`~/.config/xce/presets/<ns>/<name>.yaml`, same layout in the apiserver's presets dir), env capture by name (`pass:`; `XCE_*` and secret-looking names excluded); the apiserver only shares admin-deployed presets read-only (`GET /xce/v1/presets[/{ns}/{name}]`, files in `<TopDir>/config/presets/` or `presets.dir`, re-read on change; not in ClusterConfig), and `preset pull` copies one locally so server-side removal never affects users; on a name collision pull refuses unless it is an update of the same server preset (`--as` / `--force` with `.bak`); submit uses local presets only (`preset diff`, `pull --force` user-initiated); user-to-user sharing by `preset export`/`import`; presets carry no privilege and are not a configuration level; optional preset-name job label; the portal does the same expansion. Brief: `.kiro/specs/xcecli-presets/brief.md`. Dependencies: application-profile.
|
||||
- [ ] profile-usage-insight — NEW 2026-10-12 (from application-profile discussion, backlog; no brief yet). Per-profile report over recent jobs comparing the profile's defaults with actual use (requested vs. peak memory, CPU efficiency, OOM kills, pend time, runtime by variant) and suggesting better values for site presets (or a profile essential), e.g. "preset openfoam rusage mem 4000, p95 peak 1600 → suggest 2000" (Kubernetes VPA recommender precedent). **Advisory only** (user decision): the admin applies changes to site presets or `applications.yaml`; nothing tunes itself. Optional user hint ("used 1.4G of 4G requested"). Dependencies: application-profile, run usage in job records.
|
||||
- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections, `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.
|
||||
|
|
|
|||
|
|
@ -18765,6 +18765,32 @@
|
|||
"record": ".wolf/bugs/bug-1969.md",
|
||||
"gitea_issue": 400,
|
||||
"gitea_synced_status": "fixed"
|
||||
},
|
||||
{
|
||||
"id": "bug-1970",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-12",
|
||||
"status_note": "newly logged",
|
||||
"component": "apiserver",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-12T13:07:37Z",
|
||||
"last_seen": "2026-10-12T13:07:37Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "apiserver accepts rerunable, requeue/checkpoint/runtime/warning submit options, never sends them\n\n**Observed:** (source reading, 2026-10-12, at 83c6eb80)\n`POST /xce/v1/jobs/create` accepts these `JobSubmitRequest` fields\n(`src/frontend/apiserver/models/jobs.go`, struct at :636) and answers success,\nbut none of them reaches the controller:\n```go\nRerunable bool `json:\"rerunable,omitempty\"`\nWarningTimePeriod int `json:\"warning_time_period,omitempty\"`\nWarningAction string `json:\"warning_action,omitempty\"`\nRequeueErrorValues string `json:\"requeue_error_values,omitempty\"`\nRunTimeEstimation int `json:\"run_time_estimation,omitempty\"`\nMigThreshold int `json:\"mig_threshold,omitempty\"`\nChkpntPeriod int `json:\"chkpnt_period,omitempty\"`\nChkpntDir string `json:\"chkpnt_dir,omitempty\"`\nInitChkpntPeriod int `json:\"init_chkpnt_period,omitempty\"`\n```\n- `rerunable`, `warning_time_period`, `warning_action`, `init_chkpnt_period`\n are not read anywhere outside `models/`.\n- `requeue_error_values`, `run_time_estimation`, `mig_threshold`,\n `chkpnt_period`, `chkpnt_dir` are copied into `models.JobInfo`\n (`controllers/jobs.go:1214-1218`, and on restart :1013-1025), but\n `modelJobToSubmit()` (`backend/convert.go:334`) never reads them, and the\n controller's `JobSubmitData` / `JobStartInfo` (`src/schema/controller.capnp`\n :583, :314) have no field for them.\n- None is in `preCheckSubmitOptions()`'s list of refused options\n (`controllers/jobs.go:1285`), which bug-1936 added for exactly this kind of\n option (in/out/err files, asked hosts, begin/term time).\n\n**Expected:** each submit option is either passed on to the job or rejected\nwith 400 naming the option, as `preCheckSubmitOptions()` does for the others.\n**Impact:** wrong result: a user asking for automatic requeue on given exit\ncodes, rerun on host failure, periodic checkpoints or a warning signal before\nthe run limit gets a success reply and a job that does none of it.\n**Found by:** review during the `application-profile` brief discussion\n(extra v1 attributes), 2026-10-12.",
|
||||
"file": "src/frontend/apiserver/controllers/jobs.go, src/frontend/apiserver/backend/convert.go, src/frontend/apiserver/models/jobs.go",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"apiserver",
|
||||
"submit",
|
||||
"silent-drop"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1968",
|
||||
"bug-1936"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1970.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
87
.wolf/bugs/bug-1970.md
Normal file
87
.wolf/bugs/bug-1970.md
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
---
|
||||
id: "bug-1970"
|
||||
status: "open"
|
||||
status_updated: "2026-10-12"
|
||||
status_note: "newly logged"
|
||||
component: "apiserver"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-12T13:07:37Z"
|
||||
last_seen: "2026-10-12T13:07:37Z"
|
||||
occurrences: 1
|
||||
files: ["src/frontend/apiserver/controllers/jobs.go", "src/frontend/apiserver/backend/convert.go", "src/frontend/apiserver/models/jobs.go"]
|
||||
tags: ["apiserver", "submit", "silent-drop"]
|
||||
related: ["bug-1968", "bug-1936"]
|
||||
---
|
||||
|
||||
# bug-1970: apiserver accepts rerunable, requeue/checkpoint/runtime/warning submit options, never sends them
|
||||
|
||||
## Description
|
||||
**Observed:** (source reading, 2026-10-12, at 83c6eb80)
|
||||
`POST /xce/v1/jobs/create` accepts these `JobSubmitRequest` fields
|
||||
(`src/frontend/apiserver/models/jobs.go`, struct at :636) and answers success,
|
||||
but none of them reaches the controller:
|
||||
```go
|
||||
Rerunable bool `json:"rerunable,omitempty"`
|
||||
WarningTimePeriod int `json:"warning_time_period,omitempty"`
|
||||
WarningAction string `json:"warning_action,omitempty"`
|
||||
RequeueErrorValues string `json:"requeue_error_values,omitempty"`
|
||||
RunTimeEstimation int `json:"run_time_estimation,omitempty"`
|
||||
MigThreshold int `json:"mig_threshold,omitempty"`
|
||||
ChkpntPeriod int `json:"chkpnt_period,omitempty"`
|
||||
ChkpntDir string `json:"chkpnt_dir,omitempty"`
|
||||
InitChkpntPeriod int `json:"init_chkpnt_period,omitempty"`
|
||||
```
|
||||
- `rerunable`, `warning_time_period`, `warning_action`, `init_chkpnt_period`
|
||||
are not read anywhere outside `models/`.
|
||||
- `requeue_error_values`, `run_time_estimation`, `mig_threshold`,
|
||||
`chkpnt_period`, `chkpnt_dir` are copied into `models.JobInfo`
|
||||
(`controllers/jobs.go:1214-1218`, and on restart :1013-1025), but
|
||||
`modelJobToSubmit()` (`backend/convert.go:334`) never reads them, and the
|
||||
controller's `JobSubmitData` / `JobStartInfo` (`src/schema/controller.capnp`
|
||||
:583, :314) have no field for them.
|
||||
- None is in `preCheckSubmitOptions()`'s list of refused options
|
||||
(`controllers/jobs.go:1285`), which bug-1936 added for exactly this kind of
|
||||
option (in/out/err files, asked hosts, begin/term time).
|
||||
|
||||
**Expected:** each submit option is either passed on to the job or rejected
|
||||
with 400 naming the option, as `preCheckSubmitOptions()` does for the others.
|
||||
**Impact:** wrong result: a user asking for automatic requeue on given exit
|
||||
codes, rerun on host failure, periodic checkpoints or a warning signal before
|
||||
the run limit gets a success reply and a job that does none of it.
|
||||
**Found by:** review during the `application-profile` brief discussion
|
||||
(extra v1 attributes), 2026-10-12.
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: n/a: found by source reading, no binary run (host ldd reports glibc 2.40)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan, debug or release alike (Go apiserver code path, independent of the C build profile)
|
||||
- Code: scheduler-framework @ 83c6eb80-dirty
|
||||
- Deployment: any: apiserver submit path, before the controller
|
||||
|
||||
## Reproduction
|
||||
1. Deploy a single node with the apiserver (`/xce-deploy-ha localhost:9991`
|
||||
plus the apiserver), log on, and keep the token in `$TOKEN`.
|
||||
2. `curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"queue":"normal","command":"exit 99","requeue_error_values":"99","rerunable":true,"chkpnt_period":5}' http://localhost:8080/xce/v1/jobs/create`
|
||||
3. Observe a success reply; the job exits with 99 and goes to EXIT (no
|
||||
requeue); `xcecli jobs list -l` shows no requeue, rerun or checkpoint
|
||||
setting.
|
||||
4. Static check: `grep -rn 'RequeueErrorValues\|Rerunable\|ChkpntPeriod' src/frontend/apiserver/backend/`
|
||||
prints nothing.
|
||||
- Frequency: deterministic (by construction; steps 1-3 not run live in this
|
||||
session, step 4 run 2026-10-12)
|
||||
- Evidence: not kept (source reading; file:line references above)
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue