docs(steering): resource log: data, cache and communication topology between jobs (PR-11)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/steering/resource-questions.md

@@ -151,6 +151,26 @@ silently.
 - **Affects:** `resource-instances` (level 1), `parjob-placement` (levels 2 and 3, locality), the resreq syntax (`cu[]` or an extended `same[]`), `gang-groups` (demote), the scheduler reservation and backfill work.
 - **Open:** see N21–N25.
 
+### PR-11: data, cache and communication topology between jobs (status: discussing)
+- **User (2026-10-11):** how about data cache, data and communication topology between jobs?
+- **One topology model:**
+  - Hosts (PR-4) carry hierarchical labels: host ⊂ leaf switch ⊂ block/spine ⊂ rack ⊂ zone. Storage pools carry where they are reachable from, and their speed.
+  - The scheduler derives one distance: same host < same switch < same block < anywhere. It serves within-job topology (PR-10 level 2), between-job placement and data locality alike.
+  - Labels first, not a full graph (as in R5). Admin config first; discovery (`ibnetdiscover`, LLDP) later through a reporter.
+- **1. Data and cache locality between jobs** (producer to consumer through files):
+  - Dataset copies and node-local caches are supply-map entries with locations (PR-5, PR-8). A consumer prefers the closest copy (Spark locality levels: node, switch, storage pool, any).
+  - **Delay scheduling** (Spark `locality.wait`): wait briefly for a near host, then relax. The same policy as N22, timed in session time so simulation stays deterministic.
+  - **Move job or move data:** place far, or stage a copy (PR-3), decided by size and lead time. Start with "prefer locality, stage explicitly".
+- **2. Communication between jobs** (RL trainer and rollouts, prefill and decode, streaming producer and consumer, services and heavy clients):
+  - **Placement groups** (like AWS cluster placement groups): a named group that jobs join (`--placement-group <name>`, or set by a profile).
+  - **Modes:** `pack` (same or nearest domain) or `spread` (fault tolerance; replica anti-affinity for `service-replica-set`).
+  - **Strictness:** hard (`required`) or soft (`preferred`), like Kubernetes inter-pod affinity with `topologyKey`.
+  - **Co-placed, not co-started:** covers most RL and prefill/decode needs without a multi-job gang (supports N24).
+  - **Caveat:** later members are tied to the first member's domain and may wait. Optionally the group reserves domain capacity (N25).
+- **3. Scheduler:** soft preferences become host scoring in the scheduler framework's host iterator; hard ones become filters. Both read only the cached snapshot (other jobs' allocations, labels, copy locations), so the scheduler stays deterministic and never calls a provider. Groups are cheaper than arbitrary job-to-job affinity.
+- **Affects:** `scheduler-framework` (scoring and filters), `parjob-placement` (shared topology labels), `service-replica-set` (spread), PR-4 (labels on host entries), PR-5/PR-8 (copy locations), `custom-resource-reporting` (topology discovery), N22/N25.
+- **Open:** see N26–N30.
+
 ## Gang use cases (2026-10-11, for N24)
 - **Two kinds of gang:**
   - **Inside one job:** all tasks of a parallel job start together. XCE already does this, and compound resreqs mix role shapes in one job (like Slurm heterogeneous jobs).
@@ -262,9 +282,15 @@ Source: the CSI specification, from general knowledge; to be checked against the
 | N23 | PR-10: rank ordering: does the scheduler return hosts in topology order, and is that order part of the persisted allocation and shown to the launcher (hostfile, env)? | open | |
 | N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | leaning | Leaning yes. Use-case review (2026-10-11), below "Gang use cases": a multi-job gang is needed mainly for RL post-training, disaggregated serving and streaming pipelines. All three mix batch and service components, so re-scope it as a "co-start group" across `service-replica-set` and jobs when RL or disaggregated serving comes onto the roadmap. |
 | N25 | PR-10: reservation or plan-ahead for large topology-constrained jobs, to prevent starvation and fragmentation: a new spec, or part of an existing one? | open | |
+| N26 | PR-11: placement groups only, or also direct job-to-job affinity (`affinity[job(...)]`)? | open | Leaning groups first (cheaper, simpler to explain). |
+| N27 | PR-11: which modes and strictness in v1: pack and spread, hard (`required`) and soft (`preferred`)? | open | |
+| N28 | PR-11: delay scheduling for data locality: wait per job, queue or profile; default? | open | Same mechanism as N22. |
+| N29 | PR-11: where topology comes from: admin config only in v1, discovery (`ibnetdiscover`, LLDP) later via a reporter? | open | |
+| N30 | PR-11: data distance: simple locality tiers, or a size-weighted cost deciding between staging a copy and placing far? | open | Leaning tiers in v1. |
 | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). |
 
 ## History
+- 2026-10-11: user asked about data cache, data and communication topology between jobs → PR-11 (one topology model; data locality with delay scheduling; placement groups with pack or spread); N26–N30 added.
 - 2026-10-11: user asked for the use cases of gangs → "Gang use cases" section; N24 leaning (demote `gang-groups`, re-scope later as a co-start group for RL, disaggregated serving and streaming).
 - 2026-10-11: user asked whether LLM training can be a single job with complex topology requirements → PR-10 (single parallel job as the default; topology at three levels; `gang-groups` optional); N21–N25 added.
 - 2026-10-11: user input: keep submit options simple → PR-7 (N6 leaning); data producer/consumer dependencies → PR-8 (N16–N19); isolation between users, sharing low priority → PR-9 (N20); requeue/restart reuse the previous directory, details later → N11/N13 leaning.