myxce/.kiro/specs/scheduler-allocation/research.md

## Summary
- **Feature**: `scheduler-allocation`
- **Discovery Scope**: Extension (existing system) — light discovery per `design-discovery-light.md`. The bulk of architectural discovery already happened during an extended, adversarial requirements-review pass (documented inline in `requirements.md`, not repeated here); this log covers the additional, implementation-level verification done specifically to unblock design.md.
- **Key Findings**:
  - `kv.c`'s real `kv_key_ops_t.cmp` contract is a three-way comparator (0 = equal), confirmed against `cmp_int_key`/`kvtab_get`'s `cmp(...) == 0` check — `requirements.md`'s illustrative `cache_key_cmp_()` had the inverted (boolean) polarity and has been corrected in place.
  - `sel_key_assign_fn` (`xce_select.c`) is a live precedent for exactly the struct-copy-on-insert pattern `cache_key_ops.assign` needs — same shape, confirms the approach rather than inventing one.
  - The scheduler's `sched_core_isolation_lib` (zero-controller-I/O, mechanically tested) does not constrain this feature: `xce_alloc_plan_iter_t`/`xce_host_iter_t`/the cache are pure computation with no I/O, and live in `scheduler.c`/new plugin files — outside that isolated set — so no invariant conflict.

## Research Log

### `kvtab_t`'s real key-ops contract (vs. requirements.md's illustrative pseudocode)
- **Context**: Requirements review flagged that `requirements.md`'s Req 7 cache pseudocode used an invented `kvtab_lookup(tab, key, hash_fn, eq_fn)` free function and an unverified `.cmp` return convention — both needed confirming against the real `src/lib/kv.c`/`kv.h` before design could treat them as implementable.
- **Sources Consulted**: `src/lib/kv.h` (`kv_key_ops_t`, `kv_params_t` declarations), `src/lib/kv.c` (`kvtab_get`, `kvtab_add`, `cmp_int_key`, `int_key_ops`, `string_key_ops`), `src/lib/xce_select.c` (`sel_key_assign_fn`, an existing struct-key precedent).
- **Findings**:
  - A `kvtab_t` binds exactly one `kv_key_ops_t` (`.hash`/`.assign`/`.cmp`/`.destroy`) at table-creation time via `kv_params_t`; lookups/inserts go through generic `key_find()`/`key_add()`/`key_del()`, never a per-call hash/eq pair.
  - `cmp_int_key()` (`int_key_ops.cmp`) returns `-1`/`0`/`1` (a `strcmp`-style three-way comparator), and `kvtab_get()` checks `t->keyops->cmp(e->key, key) == 0` for a match. `requirements.md`'s original `cache_key_cmp_()` returned a boolean (1 for match) — inverted. **Fixed in place** during this discovery pass (see requirements.md's own inline correction note).
  - `sel_key_assign_fn()` (`xce_select.c:113-122`) is a real, already-shipped `.assign` implementation for a composite struct key (`sel_node_key_t`): `malloc` + `memcpy` + `*dst = copy`, returning `1` on success. `requirements.md`'s `cache_key_assign_()` returns `0` on success instead — functionally equivalent (`kvtab_add()` only checks `rc < 0` as failure) but a one-line style deviation from precedent, not worth changing.
  - Only `int_key_ops`/`string_key_ops` exist as pre-built `kv_key_ops_t` instances; `cache_key_ops` (Req 7) is a legitimate third, following the same shape as `sel_node_key_t`'s.
- **Implications**: `xce_cache_key_t`'s `kv_key_ops_t` is fully specified and correct as of this pass; design.md can reference `requirements.md`'s corrected code directly rather than re-deriving it.

### Scheduler core-isolation boundary vs. this feature's new code
- **Context**: `src/scheduler/CLAUDE.md` documents a mechanically-enforced invariant (`test_core_isolation.c`) that `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` carry zero controller I/O. Needed to confirm this feature's new types/functions don't land inside that isolated set, or if they do, that they don't violate it.
- **Sources Consulted**: `src/scheduler/CLAUDE.md`, `src/scheduler/src/scheduler.c` (`sched_allocator`, `sched_job_picker`, `sched_make_*_allocation`), `src/scheduler/src/core_session.c` (`core_begin_session_internal`, `xce_core_begin_session`/`xce_core_signal_go`/`xce_core_end_session`).
- **Findings**:
  - This feature's new code (`xce_alloc_plan_iter_t`, `xce_host_iter_t`, `parjob_allocator`/`seqjob_allocator` bodies, the unsatisfiable-resreq cache and its `kv_key_ops_t`) lives in `scheduler.c` and `src/plugins/*.c` — outside the four isolated-core files.
  - The one exception is the cache's `begin_session` clear, which is placed inside `core_begin_session_internal()` (`core_session.c`) — already inside the isolated set. This is safe: the clear touches only the cache's own in-memory kvtab, no I/O, matching the existing `core_reset_shared_resval()`/`core_reset_workers()` calls already living in that same function with zero locking.
  - `xce_core_begin_session()`'s own doc comment (`core_session.c:359-368`) already documents the happens-before property this feature's cache-clear placement depends on (driver completes all begin-session reset work strictly before `xce_core_signal_go()` unblocks the picker/allocator) — no new synchronization primitive needed.
- **Implications**: No core-isolation conflict. Design.md's Boundary Commitments section cites this directly rather than re-deriving it (per design-review-gate.md's "Project Invariant Cross-Check").

### Test file / build registration convention
- **Context**: File Structure Plan needs concrete new test file paths and their meson wiring.
- **Sources Consulted**: `src/scheduler/testsuite/meson.build` (existing `executable()`/`test()` block pattern, `scheduler_test_deps`).
- **Findings**: Every test unit is its own `executable()` (sharing `scheduler_test_deps`) plus a matching `test()` registration in the `scheduler` suite — one block per `test_<unit>.c` file, following `structure.md`'s stated convention (`test_<unit>.c` naming, `testsuite/` location).
- **Implications**: New test files (plan iterator, host iterator, cache, parjob combinatorial, seqjob-rewrap equivalence) each get their own `executable()`/`test()` block, not folded into an existing one.

## Architecture Pattern Evaluation

Already resolved in `requirements.md`'s own "Alternatives Considered" section (Approach A: eager pre-computed plan array, rejected for memory pressure and no reusable abstraction; Approach B: lazy iterator composition, chosen). Not re-litigated here — see `requirements.md` brief for the full comparison table and rationale.

## Design Decisions

### Decision: Where the unsatisfiable-resreq cache's `begin_session` clear lives
- **Context**: Req 7 AC 2 requires the cache cleared every scheduling session, correctly, without a race against the allocator thread reading/writing it.
- **Alternatives Considered**:
  1. A new, independent hook (e.g. its own plugin `begin_session_fn`) — adds a synchronization question that doesn't already have a proven answer.
  2. Wire the clear directly into `core_begin_session_internal()`, alongside the existing `core_reset_shared_resval()`/`core_reset_workers()` calls.
- **Selected Approach**: (2).
- **Rationale**: `core_begin_session_internal()` already resets other shared, per-session state with zero explicit locking, relying on `xce_core_begin_session()`'s documented ordering guarantee (reset work completes strictly before `xce_core_signal_go()` unblocks the picker/allocator). Reusing this exact placement gets the same safety property for free instead of inventing a new one.
- **Trade-offs**: Couples the cache's lifecycle to `core_session.c`, a file outside this feature's otherwise-isolated new code — acceptable, since it's a one-line addition next to an established, analogous pattern, not new architecture.
- **Follow-up**: None — verified against live source during requirements review, not merely assumed.

### Decision: `xce_alloc_plan_t` as a bucket-list partition, not `(nhosts, slots_per_host)`
- **Context**: The original brief/requirements draft modeled a plan as a uniform `(nhosts, slots_per_host)` pair, which cannot express uneven splits (e.g. 3 tasks across 2 hosts as `[2,1]`).
- **Alternatives Considered**:
  1. Uniform pair (original) — simple, but structurally cannot represent the general case Req 4 requires.
  2. Bucket-list partition (`nbuckets` + `bucket_slots[]`) — generalizes correctly; v1 generates exactly one canonical (balanced ceil/floor) partition per host-count, bounding plan count to O(N).
- **Selected Approach**: (2).
- **Rationale**: Req 4's own worked examples (`N=3` → `[3]`, `[2,1]`, `[1,1,1]`) are not representable under (1). Balanced-partition-only generation keeps v1's plan count linear in `maxTasks` rather than the full integer-partition space.
- **Trade-offs**: Slightly more complex struct than a pair, but the complexity is inherent to the problem, not incidental.
- **Follow-up**: None.

### Decision: `host_iter` is re-created fresh per plan bucket, never shared across a plan's buckets
- **Context**: A shared, forward-only `host_iter` across all buckets of a multi-bucket plan permanently drops a host an earlier (larger) bucket tried and rejected as too small, even when it exactly satisfies a later, smaller bucket — found live during design discussion by tracing a concrete worked example (`plan=[2,1]`, host A with 1 free slot, host B with 2).
- **Alternatives Considered**:
  1. One `host_iter` per plan, shared across buckets (original, incorrect) — silently fails solvable multi-host allocations.
  2. A fresh `host_iter` instance per bucket, `is_claimed()` as the only cross-bucket exclusion.
- **Selected Approach**: (2).
- **Rationale**: Only a host actually *claimed* by an earlier bucket should be excluded from later buckets; a host merely *offered and rejected* (too small) must remain eligible. Only a fresh per-bucket scan can express this correctly, since a forward-only cursor cannot "reconsider" a previously-rejected candidate.
- **Trade-offs**: One more `init()`/`fini()` pair per bucket than the (incorrect) shared-cursor version — a fixed, small constant-factor cost per bucket, not a complexity-class change.
- **Follow-up**: None — this is now Req 4 AC 2's own explicit language, not just an implementation detail.

## Risks & Mitigations
- **`parjob_prepare_allocation()`/`seqjob_prepare_allocation()` fan-out corruption** — `plugin_prepare_allocation()` unconditionally calls every registered plugin's `prepare_allocation_hook`, and `xvec_idval_store()` has no dedup by resource idx. Fixing `parjob_prepare_allocation()` without also gating `seqjob_prepare_allocation()` on `maxTasks == 1` would append a second, wrong resource-want entry for every parjob allocation attempt. Mitigation: both changes land together (Req 4 AC 6), never one without the other.
- **`resreq_fp` cold on snapshot-restore sessions** — accepted, not mitigated: confirmed `xce_sched_core_load_snapshot()`'s 12 call sites are all in `sim.c` (simulation/investigation tooling), never `driver.c`, so this can only ever cost cycles in an offline debugging session, never production throughput. Tracked in bug-1767 for future cleanup, deliberately not a blocking requirement here.
- **`sched_make_compound_allocation()`'s missing rollback on partial COMPOUND failure** (bug-1766) — pre-existing, out of this spec's boundary (Req 9 AC 2 keeps COMPOUND/ALTER dispatch unchanged). Not fixed here; flagged so a future spec addressing it doesn't have to re-discover it.

## References
- `requirements.md` (this spec) — the primary source of truth for this design; contains the full discovery/design discussion this research log summarizes rather than duplicates.
- `.wolf/buglog.json` bug-1766, bug-1767 — pre-existing/adjacent gaps found during review, deliberately kept out of scope.
- `src/scheduler/CLAUDE.md` — core-isolation invariant, driver/picker/allocator threading model.
- `.kiro/steering/resource-accounting.md` — confirms the `available = resvals - resInUse` admission-check formula this feature's `try_alloc_on_host()` relies on, unchanged by this spec.