myxce/.kiro/specs/xcecli-presets/brief.md

# Brief: xcecli-presets

(From the `application-profile` discussion, 2026-10-12; user decisions.
Backlog.)

## Problem
Application profiles hold only essentials, what every job of an application
must have (`job-config-layering.md`, additive rule). Users still repeat the
same tuning on every job: `rusage[mem=...]`, `span[ptile=...]`, a task
count, env, the queue. They also tune these over time (sizing, placement)
and want to keep and share what they find. Putting such defaults in the
profile made the server-side rules complex (override markers per section
and per term, ranges, privileged overrides) and would add a per-user
configuration level.

## Desired Outcome
- **A preset is a set of job options** (`--app`, `--queue`, `-R`, `-n`,
  env, limits, `--app-param`, ...), named and stored by the client. `xcecli jobs submit
  --preset <name>` expands it; any command-line flag replaces the preset's
  value for that option as a whole (no resreq parsing). The submit carries
  only the expanded values.
- **Handled fully on the client.** Presets never reach the controller, WAL
  or scheduler, own no attribute, are not a configuration level and carry
  no privilege: `app:use`, `queue:submit`, the profile's essentials, queue
  ceilings and project quotas still apply at submit.
- **One file per preset** (user, 2026-10-12), the same layout on client
  and server:

  ```
  ~/.config/xce/presets/                 # client (personal + pulled)
      openfoam-large.yaml
      cfd-team/openfoam-large.yaml       # pulled from a namespace keeps its path
  <apiserver presets dir>/               # server (admin-deployed, read-only)
      openfoam.yaml
      cfd-team/openfoam-large.yaml       # served as "cfd-team/openfoam-large"
  ```
  - The file name is the preset name (no `name:` inside); names use
    `[A-Za-z0-9._-]`; namespaces are one level of subdirectory.
  - Writes are atomic (temp file + rename); a syntax error breaks one
    preset only; xcecli rewrites only the file it changes, so other files
    keep their comments.
  - `preset export a b c` writes a multi-document YAML bundle (`---`, each
    with its name); `preset import` splits it back into files.
  - `preset pull <ns>/<name>` keeps the namespace path; `--as <name>`
    renames locally.
  - Why not one `presets.yaml`: concurrent xcecli runs would overwrite each
    other, one syntax error breaks all presets, rewriting loses comments,
    and server-side add/remove becomes editing a shared file.
- **Personal presets** are files under `~/.config/xce/presets/`; users
  change them on the fly (`xcecli preset set / edit / show / list / delete`).
- **The server only shares.** The apiserver serves admin-deployed preset
  files read-only: `GET /xce/v1/presets` (list, with each preset's content
  hash) and `GET /xce/v1/presets/{name}` / `{ns}/{name}` (fetch). Every
  authenticated user may see them.
- **Where server presets live** (user, 2026-10-12): a directory beside the
  apiserver's other config, `<TopDir>/config/presets/` (as
  `config/rbac.yaml` and `config/auth.yaml`, `apiserver/cmd/main.go:144`,
  `:152`), overridable by `presets.dir` in `app.yaml` (e.g.
  `/etc/xce/presets`).
  - One file per preset; site presets at the top level, team presets one
    subdirectory deep (`cfd-team/openfoam-large.yaml`).
  - Re-read when a file's mtime changes, no restart; a file that fails to
    parse is skipped, logged and not listed, without affecting the others.
  - No database, no HA. With several apiservers the directory is deployed
    to each, like `rbac.yaml`; the content hashes in the listing let a
    client notice a mismatch.
  - Delegation needs no feature: `presets.dir` can be a git checkout pulled
    by a timer, so teams change presets by pull request with history.
  - Not in `ClusterConfig`: controller and scheduler never use presets
    (independence rule, `config-design.md`) and presets change far more
    often than cluster config (frequency rule); it would also turn every
    preset edit into a cluster config upgrade.
- **Pull copies.** `xcecli preset pull <name>` writes a complete, standalone
  local preset with an informational `origin` (server name, content hash,
  pull date). Removing or changing a server preset never affects local
  copies. `xcecli preset diff <name>` compares with the server (reports when
  it is gone); `preset pull --force` refreshes. Both are user-initiated.
- **Name collisions on pull** (user, 2026-10-12). Namespaced presets go to
  their subdirectory, so only the same path can collide:
  - the local file is a pulled copy of the same server preset (its `origin`
    matches): an update; pull shows the diff and stops, `--force` replaces
    it;
  - anything else (a personal preset, or a copy from another server or
    path): pull refuses and suggests `--as <name>` (pull under another local
    name; `origin` keeps the server name, so `diff` still works) or
    `--force`;
  - `--force` keeps the replaced file as `<name>.yaml.bak`;
  - `preset pull --all [ns]` applies the same rules per preset, listing and
    skipping collisions.
- **Submit uses local presets only.** `--preset X` not found locally but
  present on the server stops with "not found locally; the server has `X`:
  run `xcecli preset pull X`". Never an automatic pull, so server changes
  or removals never affect a submit.
- **User-to-user sharing** without the server: `preset export` / `import`
  (file, mail, git).
- **Site presets complement profiles.** Admins typically publish a preset
  with the profile's name holding its usual tuning; clients may point to it
  ("server preset `openfoam` available") but never apply it automatically.
- **Environment capture** (user, 2026-10-12). A preset may list env
  variable *names or patterns* to copy from the submit shell; values are
  read at submit, never stored in the preset (LSF `bsub -env "all, ~VAR"`,
  Slurm `--export=VAR1,VAR2` precedent):

  ```yaml
  env:
    pass: [OMP_NUM_THREADS, "MPI_*", "FOAM_*"]   # or: pass: all, exclude: ["SSH_*", DISPLAY]
    set:  { OMP_PROC_BIND: close }
  ```
  - `XCE_*` is never captured (reserved; matters for submits from inside a
    job, which today copy the parent's `XCE_JOBID` with `--env`).
  - Secret-looking names (`*TOKEN*`, `*SECRET*`, `*PASSWORD*`, `*_KEY`) are
    excluded by default with a warning, because captured values are stored
    in the job record; an explicit `set:` overrides.
  - A captured name the job's profile defines is dropped by the client with
    a note; an explicit `set:` that conflicts is still rejected by the
    server (additive rule). The client reads the profile (`app:view`).
  - `pass: all` is opt-in; no capture is the default (today's behaviour).
    CLI only: portal and REST have no shell env and use `set:` only.
  - Today's `xcecli jobs submit --env` (whole `os.Environ()`,
    `xcecli/common/utils/utils.go:279`) gets the same `XCE_*` and secret
    exclusions, so the flag and presets behave alike.
- **Safety and visibility:** a warning on export/import when env variable
  names look like secrets; `xcecli jobs submit --dry-run` shows the expanded
  options and the effective values after the profile merge
  (`job-config-merge`).
- **No preset field on the server; preset identity in `exdata.meta`**
  (user, 2026-10-12). Expansion is entirely client-side and the server sees
  ordinary job options. To compare variants, the client adds one opaque key
  to the job's exdata JSON:

  ```json
  { "sub_env_vars": {...}, "limits": {...},
    "meta": { "preset": "cfd-team/openfoam-large", "preset_hash": "3f2a9c" } }
  ```
  - exdata already round-trips with no schema change: client →
    `start_info.exdata` → WAL submit fact and snapshot → worker, and job
    queries return it as `extend_data` (`apiserver/backend/convert_pure.go`
    :388, :537); 16 KB cap (`maxExdataLen`, :395).
  - `meta` is client-only metadata: apiserver and controller pass it
    through uninterpreted (the controller's dispatch-time rewrite of exdata
    limits, `job-config-merge`, preserves it). The worker's `parse_exdata()`
    (`worker/src/xcecd_exdata.c`) ignores unknown top-level keys but logs a
    NOTICE for each; this spec makes `meta` a known, silently skipped key
    (one-line worker change).
  - `xcecli preset report <name>` lists the user's jobs through the normal
    API and filters on `extend_data.meta.preset` client-side: runtime, peak
    memory, efficiency and exit status per preset or preset version. It
    covers jobs from any login node, and the portal when it writes the same
    key. No local submission log.
  - Limits: the server cannot filter or index `meta`, so the report fetches
    and filters the user's jobs (fine for personal tuning); `meta` is
    whatever the client wrote, informational only, never trusted.
  - Server-side labels for filtering and team-wide grouping are a separate,
    general feature (`job-labels`, backlog).
- **Portal:** the same expansion (a small option-level merge) and the same
  read-only listing; its personal presets are stored by the portal itself
  and are not shared with xcecli in the first version.

## Scope
- **In:** preset format, xcecli commands, expansion and precedence, the
  read-only apiserver listing/fetch and its file directory, pull/diff,
  export/import, secret warning, dry-run display, `exdata.meta` and
  `preset report`, the worker's `meta` skip, tests.
- **Out:** writable server-side shared presets (would need owners,
  visibility and durable storage; revisit only if the portal needs it, after
  `rbac-persistence`), automatic application of presets, any server-side
  preset expansion.
- **Open:** the portal's preset storage.

## Upstream / Downstream
- **Upstream:** `application-profile` (profiles and `--app`),
  `job-config-merge` (dry-run effective values).
- **Downstream:** `profile-usage-insight` (advisory suggestions for site
  presets).