diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md index f8a41dac..e4d29e18 100644 --- a/.kiro/steering/resource-questions.md +++ b/.kiro/steering/resource-questions.md @@ -132,6 +132,25 @@ silently. - Shared writable space, if ever needed, is a project-owned directory with group permissions. - Sharing is what enables cross-user PR-8. +### PR-10: LLM training as a single parallel job with topology requirements (status: discussing) +- **User (2026-10-11):** treat LLM training as a single job with complex topology requirements. +- **Analysis:** yes, as the default model. Training is **one parallel job**, not a gang of jobs. + - **Already in XCE:** + - all-or-nothing placement of one parallel job; + - `span[ptile|hosts|block]` (`parjob-placement`); + - compound resreqs for roles of different shapes inside one job (`lib/xce_resreq.peg:60-83`), e.g. `1*{select[!gpu]} + 16*{rusage[gpu=8] span[ptile=1]}` (coordinator plus GPU workers); + - one lifecycle for kill, requeue, checkpoint and accounting. + - **Topology, at three levels:** + 1. **Inside a host:** GPU and NIC choice (NVLink domain, rail affinity, NUMA). From `resource-instances` (identified instances with attributes) and `affinity[]`, which is parsed but unused (`lib/resreq.c:127`). + 2. **Across hosts:** topology domains (leaf switch, rail group, IB block, NVL72 rack), stored as host attributes in the supply map. A request bounds and packs domains. Precedents: LSF compute units `cu[type=switch:maxcus=2:pref=minavail]` (fits XCE's LSF-style syntax), and Slurm `--switches=N@maxwait` (wait up to a limit for better topology, then relax). `parjob-placement` already notes locality scoring as later work. + 3. **Rank order:** the allocation returns hosts in topology order so adjacent ranks or pipeline stages are close; the launcher's hostfile follows it. The ordered allocation is persisted field-complete. + - **Launch:** the launcher on the first host gets the hostfile and environment (`MASTER_ADDR`, `WORLD_SIZE`, ranks) and starts torchrun, DeepSpeed or MPI, or starts remote tasks through xcecd's remote-task path. + - **Failure:** host loss fails or requeues the job; it restarts from its checkpoint claim. Elastic training comes later via `job-resize`. + - **Scheduler impact:** topology-aware selection is combinatorial and needs heuristics (best-fit domain packing). Large GPU jobs need reservation or plan-ahead so small jobs do not fragment domains and starve them. +- **Consequence:** `gang-groups` becomes optional and later. It is only for roles that need separate lifecycles, profiles or owners, e.g. an RL trainer plus rollout or inference services. +- **Affects:** `resource-instances` (level 1), `parjob-placement` (levels 2 and 3, locality), the resreq syntax (`cu[]` or an extended `same[]`), `gang-groups` (demote), the scheduler reservation and backfill work. +- **Open:** see N21–N25. + ## CSI notes (2026-10-11, for C3 and R4) Source: the CSI specification, from general knowledge; to be checked against the current spec version during design. @@ -222,9 +241,15 @@ Source: the CSI specification, from general knowledge; to be checked against the | N18 | PR-8: version numbering (`@auto` increment, timestamps, user tags) and `@latest` resolution: at submit or at dispatch? | open | | | N19 | PR-8: a consumer whose producer fails or never seals: pend forever, or fail like an unsatisfiable `job-dependency` condition? | open | Follow `job-dependency`'s rule for never-satisfied conditions. | | N20 | PR-9: sharing granularity: dataset visibility (private/project/public, read-only) is enough, or are explicit per-user grants and writable shared space needed? How is visibility enforced on the filesystem (group ACLs, read-only mounts)? | open | Low priority (user 2026-10-11). | +| N21 | PR-10: cross-host topology syntax: an LSF-style `cu[type=...:maxcus=...:pref=...]` clause, or an extension of `same[]`/`span[]`? | open | Leaning `cu[]`, because XCE's resreq follows LSF. | +| N22 | PR-10: relax policy: wait up to a limit for a better topology, then relax (Slurm `--switches=N@maxwait`)? Per job, or per queue or profile? | open | | +| N23 | PR-10: rank ordering: does the scheduler return hosts in topology order, and is that order part of the persisted allocation and shown to the launcher (hostfile, env)? | open | | +| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | open | Leaning yes. | +| N25 | PR-10: reservation or plan-ahead for large topology-constrained jobs, to prevent starvation and fragmentation: a new spec, or part of an existing one? | open | | | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). | ## History +- 2026-10-11: user asked whether LLM training can be a single job with complex topology requirements → PR-10 (single parallel job as the default; topology at three levels; `gang-groups` optional); N21–N25 added. - 2026-10-11: user input: keep submit options simple → PR-7 (N6 leaning); data producer/consumer dependencies → PR-8 (N16–N19); isolation between users, sharing low priority → PR-9 (N20); requeue/restart reuse the previous directory, details later → N11/N13 leaning. - 2026-10-11: user proposed a shared volume with per-user/per-job subdirectories (`shared_volume///{in,out,err}`) instead of a volume per job → PR-6 refinement (leaning). N9 and N10 leaning; N8 reframed; N12–N15 added. - 2026-10-11: user asked whether volume provisioning can carry job input and output → PR-6; questions N6–N11 added.