docs(steering): resource log: LLM training as one parallel job with topology requirements (PR-10)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/steering/resource-questions.md

@@ -132,6 +132,25 @@ silently.
   - Shared writable space, if ever needed, is a project-owned directory with group permissions.
   - Sharing is what enables cross-user PR-8.
 
+### PR-10: LLM training as a single parallel job with topology requirements (status: discussing)
+- **User (2026-10-11):** treat LLM training as a single job with complex topology requirements.
+- **Analysis:** yes, as the default model. Training is **one parallel job**, not a gang of jobs.
+  - **Already in XCE:**
+    - all-or-nothing placement of one parallel job;
+    - `span[ptile|hosts|block]` (`parjob-placement`);
+    - compound resreqs for roles of different shapes inside one job (`lib/xce_resreq.peg:60-83`), e.g. `1*{select[!gpu]} + 16*{rusage[gpu=8] span[ptile=1]}` (coordinator plus GPU workers);
+    - one lifecycle for kill, requeue, checkpoint and accounting.
+  - **Topology, at three levels:**
+    1. **Inside a host:** GPU and NIC choice (NVLink domain, rail affinity, NUMA). From `resource-instances` (identified instances with attributes) and `affinity[]`, which is parsed but unused (`lib/resreq.c:127`).
+    2. **Across hosts:** topology domains (leaf switch, rail group, IB block, NVL72 rack), stored as host attributes in the supply map. A request bounds and packs domains. Precedents: LSF compute units `cu[type=switch:maxcus=2:pref=minavail]` (fits XCE's LSF-style syntax), and Slurm `--switches=N@maxwait` (wait up to a limit for better topology, then relax). `parjob-placement` already notes locality scoring as later work.
+    3. **Rank order:** the allocation returns hosts in topology order so adjacent ranks or pipeline stages are close; the launcher's hostfile follows it. The ordered allocation is persisted field-complete.
+  - **Launch:** the launcher on the first host gets the hostfile and environment (`MASTER_ADDR`, `WORLD_SIZE`, ranks) and starts torchrun, DeepSpeed or MPI, or starts remote tasks through xcecd's remote-task path.
+  - **Failure:** host loss fails or requeues the job; it restarts from its checkpoint claim. Elastic training comes later via `job-resize`.
+  - **Scheduler impact:** topology-aware selection is combinatorial and needs heuristics (best-fit domain packing). Large GPU jobs need reservation or plan-ahead so small jobs do not fragment domains and starve them.
+- **Consequence:** `gang-groups` becomes optional and later. It is only for roles that need separate lifecycles, profiles or owners, e.g. an RL trainer plus rollout or inference services.
+- **Affects:** `resource-instances` (level 1), `parjob-placement` (levels 2 and 3, locality), the resreq syntax (`cu[]` or an extended `same[]`), `gang-groups` (demote), the scheduler reservation and backfill work.
+- **Open:** see N21–N25.
+
 ## CSI notes (2026-10-11, for C3 and R4)
 Source: the CSI specification, from general knowledge; to be checked against the current spec version during design.
 
@@ -222,9 +241,15 @@ Source: the CSI specification, from general knowledge; to be checked against the
 | N18 | PR-8: version numbering (`@auto` increment, timestamps, user tags) and `@latest` resolution: at submit or at dispatch? | open | |
 | N19 | PR-8: a consumer whose producer fails or never seals: pend forever, or fail like an unsatisfiable `job-dependency` condition? | open | Follow `job-dependency`'s rule for never-satisfied conditions. |
 | N20 | PR-9: sharing granularity: dataset visibility (private/project/public, read-only) is enough, or are explicit per-user grants and writable shared space needed? How is visibility enforced on the filesystem (group ACLs, read-only mounts)? | open | Low priority (user 2026-10-11). |
+| N21 | PR-10: cross-host topology syntax: an LSF-style `cu[type=...:maxcus=...:pref=...]` clause, or an extension of `same[]`/`span[]`? | open | Leaning `cu[]`, because XCE's resreq follows LSF. |
+| N22 | PR-10: relax policy: wait up to a limit for a better topology, then relax (Slurm `--switches=N@maxwait`)? Per job, or per queue or profile? | open | |
+| N23 | PR-10: rank ordering: does the scheduler return hosts in topology order, and is that order part of the persisted allocation and shown to the launcher (hostfile, env)? | open | |
+| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | open | Leaning yes. |
+| N25 | PR-10: reservation or plan-ahead for large topology-constrained jobs, to prevent starvation and fragmentation: a new spec, or part of an existing one? | open | |
 | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). |
 
 ## History
+- 2026-10-11: user asked whether LLM training can be a single job with complex topology requirements → PR-10 (single parallel job as the default; topology at three levels; `gang-groups` optional); N21–N25 added.
 - 2026-10-11: user input: keep submit options simple → PR-7 (N6 leaning); data producer/consumer dependencies → PR-8 (N16–N19); isolation between users, sharing low priority → PR-9 (N20); requeue/restart reuse the previous directory, details later → N11/N13 leaning.
 - 2026-10-11: user proposed a shared volume with per-user/per-job subdirectories (`shared_volume/<user>/<jobid>/{in,out,err}`) instead of a volume per job → PR-6 refinement (leaning). N9 and N10 leaning; N8 reframed; N12–N15 added.
 - 2026-10-11: user asked whether volume provisioning can carry job input and output → PR-6; questions N6–N11 added.