myxce/.kiro/specs/scheduler-allocation/research.md

Summary

• Feature: scheduler-allocation
• Discovery Scope: Extension (existing system) — light discovery per design-discovery-light.md. The bulk of architectural discovery already happened during an extended, adversarial requirements-review pass (documented inline in requirements.md, not repeated here); this log covers the additional, implementation-level verification done specifically to unblock design.md.
• Key Findings:
  • kv.c's real kv_key_ops_t.cmp contract is a three-way comparator (0 = equal), confirmed against cmp_int_key/kvtab_get's cmp(...) == 0 check — requirements.md's illustrative cache_key_cmp_() had the inverted (boolean) polarity and has been corrected in place.
  • sel_key_assign_fn (xce_select.c) is a live precedent for exactly the struct-copy-on-insert pattern cache_key_ops.assign needs — same shape, confirms the approach rather than inventing one.
  • The scheduler's sched_core_isolation_lib (zero-controller-I/O, mechanically tested) does not constrain this feature: xce_alloc_plan_iter_t/xce_host_iter_t/the cache are pure computation with no I/O, and live in scheduler.c/new plugin files — outside that isolated set — so no invariant conflict.

Research Log

kvtab_t's real key-ops contract (vs. requirements.md's illustrative pseudocode)

• Context: Requirements review flagged that requirements.md's Req 7 cache pseudocode used an invented kvtab_lookup(tab, key, hash_fn, eq_fn) free function and an unverified .cmp return convention — both needed confirming against the real src/lib/kv.c/kv.h before design could treat them as implementable.
• Sources Consulted: src/lib/kv.h (kv_key_ops_t, kv_params_t declarations), src/lib/kv.c (kvtab_get, kvtab_add, cmp_int_key, int_key_ops, string_key_ops), src/lib/xce_select.c (sel_key_assign_fn, an existing struct-key precedent).
• Findings:
  • A kvtab_t binds exactly one kv_key_ops_t (.hash/.assign/.cmp/.destroy) at table-creation time via kv_params_t; lookups/inserts go through generic key_find()/key_add()/key_del(), never a per-call hash/eq pair.
  • cmp_int_key() (int_key_ops.cmp) returns -1/0/1 (a strcmp-style three-way comparator), and kvtab_get() checks t->keyops->cmp(e->key, key) == 0 for a match. requirements.md's original cache_key_cmp_() returned a boolean (1 for match) — inverted. Fixed in place during this discovery pass (see requirements.md's own inline correction note).
  • sel_key_assign_fn() (xce_select.c:113-122) is a real, already-shipped .assign implementation for a composite struct key (sel_node_key_t): malloc + memcpy + *dst = copy, returning 1 on success. requirements.md's cache_key_assign_() returns 0 on success instead — functionally equivalent (kvtab_add() only checks rc < 0 as failure) but a one-line style deviation from precedent, not worth changing.
  • Only int_key_ops/string_key_ops exist as pre-built kv_key_ops_t instances; cache_key_ops (Req 7) is a legitimate third, following the same shape as sel_node_key_t's.
• Implications: xce_cache_key_t's kv_key_ops_t is fully specified and correct as of this pass; design.md can reference requirements.md's corrected code directly rather than re-deriving it.

Scheduler core-isolation boundary vs. this feature's new code

• Context: src/scheduler/CLAUDE.md documents a mechanically-enforced invariant (test_core_isolation.c) that core.c/core_events.c/core_session.c/core_clock.c carry zero controller I/O. Needed to confirm this feature's new types/functions don't land inside that isolated set, or if they do, that they don't violate it.
• Sources Consulted: src/scheduler/CLAUDE.md, src/scheduler/src/scheduler.c (sched_allocator, sched_job_picker, sched_make_*_allocation), src/scheduler/src/core_session.c (core_begin_session_internal, xce_core_begin_session/xce_core_signal_go/xce_core_end_session).
• Findings:
  • This feature's new code (xce_alloc_plan_iter_t, xce_host_iter_t, parjob_allocator/seqjob_allocator bodies, the unsatisfiable-resreq cache and its kv_key_ops_t) lives in scheduler.c and src/plugins/*.c — outside the four isolated-core files.
  • The one exception is the cache's begin_session clear, which is placed inside core_begin_session_internal() (core_session.c) — already inside the isolated set. This is safe: the clear touches only the cache's own in-memory kvtab, no I/O, matching the existing core_reset_shared_resval()/core_reset_workers() calls already living in that same function with zero locking.
  • xce_core_begin_session()'s own doc comment (core_session.c:359-368) already documents the happens-before property this feature's cache-clear placement depends on (driver completes all begin-session reset work strictly before xce_core_signal_go() unblocks the picker/allocator) — no new synchronization primitive needed.
• Implications: No core-isolation conflict. Design.md's Boundary Commitments section cites this directly rather than re-deriving it (per design-review-gate.md's "Project Invariant Cross-Check").

Test file / build registration convention

• Context: File Structure Plan needs concrete new test file paths and their meson wiring.
• Sources Consulted: src/scheduler/testsuite/meson.build (existing executable()/test() block pattern, scheduler_test_deps).
• Findings: Every test unit is its own executable() (sharing scheduler_test_deps) plus a matching test() registration in the scheduler suite — one block per test_<unit>.c file, following structure.md's stated convention (test_<unit>.c naming, testsuite/ location).
• Implications: New test files (plan iterator, host iterator, cache, parjob combinatorial, seqjob-rewrap equivalence) each get their own executable()/test() block, not folded into an existing one.

Architecture Pattern Evaluation

Already resolved in requirements.md's own "Alternatives Considered" section (Approach A: eager pre-computed plan array, rejected for memory pressure and no reusable abstraction; Approach B: lazy iterator composition, chosen). Not re-litigated here — see requirements.md brief for the full comparison table and rationale.

Design Decisions

Decision: Where the unsatisfiable-resreq cache's begin_session clear lives

• Context: Req 7 AC 2 requires the cache cleared every scheduling session, correctly, without a race against the allocator thread reading/writing it.
• Alternatives Considered:
  1. A new, independent hook (e.g. its own plugin begin_session_fn) — adds a synchronization question that doesn't already have a proven answer.
  2. Wire the clear directly into core_begin_session_internal(), alongside the existing core_reset_shared_resval()/core_reset_workers() calls.
• Selected Approach: (2).
• Rationale: core_begin_session_internal() already resets other shared, per-session state with zero explicit locking, relying on xce_core_begin_session()'s documented ordering guarantee (reset work completes strictly before xce_core_signal_go() unblocks the picker/allocator). Reusing this exact placement gets the same safety property for free instead of inventing a new one.
• Trade-offs: Couples the cache's lifecycle to core_session.c, a file outside this feature's otherwise-isolated new code — acceptable, since it's a one-line addition next to an established, analogous pattern, not new architecture.
• Follow-up: None — verified against live source during requirements review, not merely assumed.

Decision: xce_alloc_plan_t as a bucket-list partition, not (nhosts, slots_per_host)

• Context: The original brief/requirements draft modeled a plan as a uniform (nhosts, slots_per_host) pair, which cannot express uneven splits (e.g. 3 tasks across 2 hosts as [2,1]).
• Alternatives Considered:
  1. Uniform pair (original) — simple, but structurally cannot represent the general case Req 4 requires.
  2. Bucket-list partition (nbuckets + bucket_slots[]) — generalizes correctly; v1 generates exactly one canonical (balanced ceil/floor) partition per host-count, bounding plan count to O(N).
• Selected Approach: (2).
• Rationale: Req 4's own worked examples (N=3 → [3], [2,1], [1,1,1]) are not representable under (1). Balanced-partition-only generation keeps v1's plan count linear in maxTasks rather than the full integer-partition space.
• Trade-offs: Slightly more complex struct than a pair, but the complexity is inherent to the problem, not incidental.
• Follow-up: None.

Decision: host_iter is re-created fresh per plan bucket, never shared across a plan's buckets

• Context: A shared, forward-only host_iter across all buckets of a multi-bucket plan permanently drops a host an earlier (larger) bucket tried and rejected as too small, even when it exactly satisfies a later, smaller bucket — found live during design discussion by tracing a concrete worked example (plan=[2,1], host A with 1 free slot, host B with 2).
• Alternatives Considered:
  1. One host_iter per plan, shared across buckets (original, incorrect) — silently fails solvable multi-host allocations.
  2. A fresh host_iter instance per bucket, is_claimed() as the only cross-bucket exclusion.
• Selected Approach: (2).
• Rationale: Only a host actually claimed by an earlier bucket should be excluded from later buckets; a host merely offered and rejected (too small) must remain eligible. Only a fresh per-bucket scan can express this correctly, since a forward-only cursor cannot "reconsider" a previously-rejected candidate.
• Trade-offs: One more init()/fini() pair per bucket than the (incorrect) shared-cursor version — a fixed, small constant-factor cost per bucket, not a complexity-class change.
• Follow-up: None — this is now Req 4 AC 2's own explicit language, not just an implementation detail.

Risks & Mitigations

• parjob_prepare_allocation()/seqjob_prepare_allocation() fan-out corruption — plugin_prepare_allocation() unconditionally calls every registered plugin's prepare_allocation_hook, and xvec_idval_store() has no dedup by resource idx. Fixing parjob_prepare_allocation() without also gating seqjob_prepare_allocation() on maxTasks == 1 would append a second, wrong resource-want entry for every parjob allocation attempt. Mitigation: both changes land together (Req 4 AC 6), never one without the other.
• resreq_fp cold on snapshot-restore sessions — accepted, not mitigated: confirmed xce_sched_core_load_snapshot()'s 12 call sites are all in sim.c (simulation/investigation tooling), never driver.c, so this can only ever cost cycles in an offline debugging session, never production throughput. Tracked in bug-1767 for future cleanup, deliberately not a blocking requirement here.
• sched_make_compound_allocation()'s missing rollback on partial COMPOUND failure (bug-1766) — pre-existing, out of this spec's boundary (Req 9 AC 2 keeps COMPOUND/ALTER dispatch unchanged). Not fixed here; flagged so a future spec addressing it doesn't have to re-discover it.

References

• requirements.md (this spec) — the primary source of truth for this design; contains the full discovery/design discussion this research log summarizes rather than duplicates.
• .wolf/buglog.json bug-1766, bug-1767 — pre-existing/adjacent gaps found during review, deliberately kept out of scope.
• src/scheduler/CLAUDE.md — core-isolation invariant, driver/picker/allocator threading model.
• .kiro/steering/resource-accounting.md — confirms the available = resvals - resInUse admission-check formula this feature's try_alloc_on_host() relies on, unchanged by this spec.