myxce/.kiro/specs/scheduler-sim-live-allocation/research.md

Research & Design Decisions

Summary

• Feature: scheduler-sim-live-allocation
• Discovery Scope: Extension (existing system) — light discovery per design-discovery-light.md, since the integration points, existing patterns, and constraints were already exhaustively traced against live source during requirements development (see requirements.md's Project Description for the full evidence trail; this document focuses on decisions, alternatives, and risk, not re-deriving what's already cited there).
• Key Findings:
  • Six functions (plugin_begin_session, plugin_end_session, sched_resolve_job_picker, sched_get_job, sched_account_jobs, sched_commit_job_alloc) already completed the exact xce_scheduler_t* → xce_sched_core_t* migration this spec needs to repeat for the remaining allocation chain — a proven, six-times-repeated pattern, not a new one.
  • Every function in the host-scoped allocation chain (sched_meet_host_selection, sched_check_shared_satisfy, sched_check_host_satisfy, plugin_prepare_allocation, sched_try_alloc_on_host_, sched_commit_decision, plugin_commit_decision) only ever touches sched->core.* or worker/ctx fields — zero genuine xce_scheduler_t-specific dependency, confirmed by reading every function body, not inferred from signatures.
  • The one genuine exception, xce_scheduler_t.unsatisfiable_cache, is a pure performance optimization with a documented prior memory-corruption incident (scheduler-allocation task 5.2 round 1) from trying to shortcut it via a struct-prefix cast — the fix pattern (a driver-registered callback, cache_clear_cb) is proven and must be repeated, not re-invented.
  • sched_allocator()'s full body (verified, not assumed) shows every single call it makes becomes core-safe once the above migration lands — the function itself needs only its top cast changed (xce_scheduler_t* sched → xce_sched_core_t* core).
  • Sim mode's thread-spawn point is sched_sim_driver_run() (sim_loop.c), NOT sched_sim_driver_start() (sim.c) — the latter only does core init and snapshot/replay-file setup; the pthread_create(&picker_tid, NULL, sim_picker_thread_fn, &(sim->core)) call lives inside the former.

Research Log

Is the allocation chain's xce_scheduler_t* coupling mechanical or genuine?

• Context: The spec's central bet is that sim mode can reuse normal mode's allocation code verbatim. This only holds if the coupling is superficial (never-migrated signature) rather than load-bearing (genuinely needs normal-mode-only state).
• Sources Consulted: Direct reads of resource.c (sched_meet_host_selection:221, sched_check_shared_satisfy:305/sched_check_host_satisfy:378/sched_try_alloc_on_host_:497), plugin.c (plugin_prepare_allocation:591, plugin_commit_decision:442, plugin_job_election:554), scheduler.c (sched_commit_decision:1607, sched_allocator:429), schedobj.c (sched_ctx_get_uncomit_host:522, already core-safe).
• Findings: Every one of the 8 functions in the allocation chain reads/writes only sched->core.* fields (or worker/ctx fields passed in separately) — no function reads sched->apictx, sched->cluster_cfg, sched->unsatisfiable_cache, or any other genuinely xce_scheduler_t-only field, with the single exception noted below.
• Implications: The migration is safe to do mechanically, function by function, following the exact precedent of the six already-migrated functions. No redesign of the allocation algorithm itself is needed.

Is there a genuinely xce_scheduler_t-only dependency anywhere in the chain?

• Context: A prior investigation (scheduler-allocation spec, task 5.2) hit exactly this problem once already, for a different field.
• Sources Consulted: scheduler-allocation's own Implementation Notes (task 5.1, 5.2), core.h's xce_core_cache_clear_cb_t doc comment.
• Findings: xce_scheduler_t.unsatisfiable_cache (a fingerprint → pend-reason-category cache, populated when sched_make_primary_alloc_() exhausts all candidates for a job, consulted before attempting allocation again for an identically-shaped job) has no sim-mode equivalent field, by deliberate design (Req 5 AC 17 simulation-isolation boundary from scheduler-core). A first attempt to bridge this via __attribute__((weak)) + a cast of the shared xce_sched_core_t core prefix back to xce_scheduler_t* silently corrupted xce_sched_sim_driver_t.alt_events_file (same byte offset as the cache field) — invisible to ASAN and the whole test suite until caught by review.
• Implications: This one field needs a callback seam, not a cast, mirroring cache_clear_cb exactly. Consult/populate calls must go through a NULL-checked function-pointer pair the normal-mode driver registers and sim mode leaves NULL. A NULL seam means "always take the slow path" (recompute fully), which is correct behavior, not degraded correctness — see Design Decision below.

Where does sim mode actually spawn its picker/allocator thread?

• Context: Needed to know precisely which function's thread-spawn code changes.
• Sources Consulted: sim.c's sched_sim_driver_start() (full body read) and sim_loop.c's sched_sim_driver_run() (partial body read, confirmed pthread_create call site at line ~974).
• Findings: sched_sim_driver_start() only initializes sim->core and opens/validates the replay file and snapshot — it spawns no threads. sched_sim_driver_run() is the actual long-running entry point (design.md's own "literal SimDriver entry point" per its header comment) and is where the picker thread is created, once, for the process's sim-session lifetime.
• Implications: The unification touches sched_sim_driver_run()'s thread-spawn call, not sched_sim_driver_start().

Open question not fully resolved — flagged as a risk, not asserted either way

• Context: sched_register_plugins() (scheduler.c:596) also calls register_seqjob_allocator(sched) / register_parjob_allocator(sched), and xce_sched_plugin_t has a job_alloc_fn field (plugin_job_allocate_fn_t) alongside job_elect_fn. This raises the question of whether primary-resreq allocation is ALSO dispatched through a generic plugin-hook indirection (a plugin_job_allocate() analogous to plugin_job_election()) that would need the same core-safety migration.
• Findings so far: The scheduler-allocation spec's own Implementation Notes describe sched_make_primary_alloc_() as directly branching between seqjob_allocator.c/parjob_allocator.c's own my_alloc() functions by name (based on primary->maxTasks), which reads as a direct function-call dispatch, not a generic plugin-hook indirection — suggesting job_alloc_fn/register_seqjob_allocator/register_parjob_allocator may be a legacy/inactive registration path (parallel to job_elect_fn's own documented legacy-compat status for pickers) rather than something on the live call path.
• Not confirmed with certainty. Flagged in Risks below as a design-time verification item for whoever implements task work in this area — if sched_make_primary_alloc_() turns out to dispatch through a generic hook after all, that hook needs the identical core-safe treatment as plugin_job_election/plugin_prepare_allocation/plugin_commit_decision.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Unify picker/allocator (selected)	Re-type sched_job_picker()/sched_allocator() to xce_sched_core_t*, spawn the identical functions from both drivers	Single implementation, cannot drift (directly satisfies Requirement 5); reuses 6x-proven migration pattern	Sim mode gains a real two-thread jbuf hand-off for the first time — needs the 3 named regression tests re-verified	Selected: matches the project's own stated governing principle and its documented drift-prevention rationale
Behavioral-equivalence only (rejected)	Keep sim_picker_thread_fn() as a separate implementation, extend it to call the same core-safe sub-functions, verify equivalence via tests	Smaller diff, no jbuf behavior change in sim mode	Reintroduces exactly the two-implementations-must-independently-stay-in-sync failure mode this project has already hit three times (bug-1611, bug-1613, bug-1681)	Rejected per Requirement 5 and the user's explicit direction during spec discussion
Recording-authoritative (rejected)	Keep sim_reconcile_unaccounted_jobs() as the general-case decision source; new election/allocation logic only populates side effects (limit vres) for an already-known outcome	Smallest behavior change; no live/recorded divergence handling needed	Contradicts the project's own driver-equivalence invariant (decisions should be independently reproduced, not merely replayed) and doesn't scale to future plugin hooks with other side effects	Rejected during requirements review — see requirements.md Requirement 3 AC3-5

Design Decisions

Decision: Live election/allocation is authoritative; recorded-decision reconciliation narrows to a timing-gap safety net

• Context: Once sim mode can independently compute real decisions, two sources exist for a job's outcome: the live picker/allocator, and sim_reconcile_unaccounted_jobs()'s trust-the-recording reconstruction.
• Alternatives Considered:
  1. Recording stays authoritative always; live path only supplies side effects — rejected (see table above).
  2. Live path authoritative, recording is a fallback only for jobs it hasn't reached in time — selected.
• Selected Approach: The live picker/allocator decides for any job it reaches. sim_reconcile_unaccounted_jobs()'s existing job->accounts == NULL trigger guard already means it only acts on jobs the live path hasn't yet processed — no new guard needed, this narrowing falls out of the existing idempotency check once the live path starts actually producing outcomes.
• Rationale: Matches scheduler-driver-equivalence.md's own state-equivalence invariant (decisions should be independently reproduced by the core state machine, not replayed from a trusted log) and the project's explicit governing principle for this spec.
• Trade-offs: A live-computed decision that disagrees with the recording is now possible and must be surfaced (Requirement 3 AC5) rather than silently reconciled either way — this is by design: a divergence is a genuine finding (a migration defect, or a previously-hidden issue the old fake-PEND behavior could never have exposed).
• Follow-up: Verify via --sim-verify against real recorded sessions (Scope item 9 in requirements.md) before shipping.

Decision: unsatisfiable_cache becomes an optional, driver-registered callback pair, not a relocated field

• Context: Need the primary-allocation dispatch chokepoint to remain core-safe without giving sim mode a cache it structurally cannot support cleanly today.
• Alternatives Considered:
  1. Move the cache itself onto xce_sched_core_t so both drivers have it — rejected: sim mode has no natural place to construct/own it (no sched_fini()-equivalent teardown exists in sim mode to destroy it), and giving sim mode a working cache is not something any requirement asks for (explicitly a non-goal — see Boundary Context, "achieving simulation mode's allocation-decision throughput or latency parity").
  2. A NULL-safe callback pair (lookup/populate) on xce_sched_core_t, registered only by normal mode's main.c, mirroring cache_clear_cb — selected.
• Selected Approach: xce_sched_core_t gains two optional function-pointer fields (unsatisfiable_lookup_cb/unsatisfiable_populate_cb), each taking xce_sched_core_t* core directly — the exact same shape as the existing cache_clear_cb field, not a new wrapper struct with an opaque impl/user-data pointer (an earlier design draft added one; removed as unnecessary indirection with only one registrant and no foreseeable second). Normal mode's main.c registers both, pointing at two small adapter functions in alloc_cache.c (alongside the existing xce_alloc_cache_clear_cb() adapter) that safely recover xce_scheduler_t* via (xce_scheduler_t*)core — safe specifically because these adapters are normal-mode-exclusive and only ever invoked with a core genuinely embedded in an xce_scheduler_t, unlike the round-1 mistake which cast inside code shared by both drivers. Sim mode leaves both NULL. sched_make_primary_alloc_() treats NULL as "no cache available — always compute fresh," which is correct (never wrong), not a degraded-correctness mode.
• Rationale: Directly reuses the proven pattern and its documented rationale (core.h's xce_core_cache_clear_cb_t doc comment); avoids repeating the exact class of memory-corruption bug the earlier cast-based attempt caused.
• Trade-offs: Sim mode never gets the cache's performance benefit — accepted, explicitly out of scope (see Boundary Context).

Risks & Mitigations

• Decision-timing/ordering shift in sim mode once jbuf is genuinely used — Mitigation: re-run test_repl_equivalence.c, test_sim_contention_parity.c, test_sim_equivalence.c before considering this feature done; treat a broken assertion as a signal to investigate (it may be asserting on an artifact of the old single-fused-thread sequencing), not a reason to revert to two implementations.
• Unconfirmed job_alloc_fn/register_seqjob_allocator/register_parjob_allocator dispatch path — Mitigation: verify at implementation time whether sched_make_primary_alloc_() dispatches through a generic plugin hook or calls seqjob/parjob directly; if a hook exists, it needs the same core-safe migration as the other four plugin-hook dispatchers.
• A live-computed decision may legitimately diverge from a real recording during validation (Scope item 9) — Mitigation: this is treated as expected and valuable, not a blocker; root-cause every divergence found against real recordings before shipping, per Requirement 3 AC5.
• plugin->sched (bare void*, set only by plugin_register()) staying NULL for sim-registered plugins — Mitigation: extend the limit_seed_config()/limit_seed_config_impl() honest-core-parameter pattern (already proven by bug-1825) to the job-election and prepare-allocation call chains, rather than relying on plugin->sched.

References

• .kiro/steering/scheduler-driver-equivalence.md — the governing state-equivalence invariant.
• .wolf/buglog.json bug-1611, bug-1613, bug-1681, bug-1825, bug-1826, bug-1827 — the evidence trail of this project's own recurring drift failure mode and its prior fixes.
• .kiro/specs/scheduler-allocation/ — completed spec whose task 5.2 Implementation Notes document the unsatisfiable_cache memory-corruption incident and the cache_clear_cb fix pattern being reused here.
• src/scheduler/CLAUDE.md — "one core, two consumers" architecture statement.