Commit graph

1029 commits

Author SHA1 Message Date
f6cab0ab87
docs(spec): job-time-window discovery brief (begin/end time, runtime estimate)
New spec from /kiro-discovery (2026-09-29), added to roadmap.md:
- Begin time: the scheduler won't start a job before it (gate in
  sched_job_is_ready() using the core clock). New pend reason
  BEGIN_TIME_NOT_REACHED.
- End time: the worker enforces it for running jobs (SIGTERM, grace, SIGKILL)
  and records a persisted END_TIME_REACHED exit reason. The primary is not
  involved. A pending job whose end time can't be met stays PEND with
  END_TIME_UNMEETABLE until the user kills or modifies it.
- Runtime estimate (-We): a scheduling hint only. It sets the rule
  now + (estimate, else run limit, else 0) > end; an estimate larger than the
  run limit is rejected.
- Times and estimate can be changed only while the job is pending, through a
  new `xce jobs modify` in xcecli and a new `jobmod` in the harness. Harness
  jobsub gains all four options.
- Fixes the silent drop of xcecli -b/-t/-W and REST run_time_estimation,
  which the apiserver's modelJobToSubmit never passes to the controller.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:40:52 +00:00
7fe9b22381
docs(spec): discovery briefs for job status detail, job cleanup, sandbox execution and RBAC persistence
New specs from /kiro-discovery (2026-09-28/29), added to roadmap.md:
- job-status-detail-display: show pending reasons and reserved-vs-used
  resources in REST and xcecli. Requirements generated (7 requirements).
- scheduler-alter-pend-reasons: keep a pending reason for every alternative
  resreq (per-alternative JobPendReason) and fix the obj->reason leak.
- controller-job-cleanup: primary-only sweep that removes finished jobs past
  a configurable retention, via a replicated WAL removal fact.
- application-profile: admin-defined application profiles (execution mode,
  launch type, env, resource defaults) chosen per job, with app:use/app:view
  RBAC. Queues and profiles stay orthogonal.
- job-sandbox-execution / job-sandbox-multinode: run jobs in a systemd
  DynamicUser sandbox behind a pluggable execution backend.
- rbac-persistence: make RBAC changes made through the API survive an
  apiserver restart (closes the security.md gap).

roadmap.md also records the scheduler-sim-diagnostics info job update and a
security item: remote-task units appear to run as root (found by code
reading).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 08:57:31 +00:00
b283f3b93c
docs(spec): scheduler-allocator-runner and scheduler-sim-decision-trace requirements
scheduler-allocator-runner: brief and requirements for moving the duplicated
seqjob/parjob my_alloc() body into one framework-owned allocator runner.

scheduler-sim-decision-trace: brief and requirements for a per-job decision
trace in the simulator REPL (converged after 9 review rounds). The
decision-reasoning seed capability in scheduler-sim-diagnostics now points to
this spec.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 08:57:23 +00:00
674b29c951
chore(gitea-sync): record Gitea sync state for 4 specs and bug-1878..1882
Specs: scheduler-allocator-runner (#67: 317-319), scheduler-sim-decision-trace
(#68: 320, 324, 325), job-status-detail-display (#69: 321-323),
scheduler-sim-discrete-clock (#70: 326-328). Bugs: #329-#333.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 08:01:05 +00:00
cdc7b46d2d
chore(spec): libxce-alloc-preason-equality implementation complete (validation GO)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 07:02:49 +00:00
4e57bde693
feat(libxce-alloc-preason-equality): replay reproduces the reservation start time (task 5.3)
Add test_sim_replay_reproduces_recorded_alloc_from: a recorded DISPATCH at
an epoch-scale beginRound with a sub-second part replays at the default
simulation settings with no --sim-verify mismatch, and the live allocation's
from equals beginRound in whole seconds. Share the dispatch fixture setup
of the existing tests through two helpers.

Live: 2-round HA soak recordings (11 DISPATCH decisions) replayed with
--sim-verify and no --sim-speed report 0 mismatches.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 06:48:38 +00:00
9fae9ca22a
chore(spec): scheduler-sim-discrete-clock implementation complete (validation GO); unblock libxce-alloc-preason-equality 5.3
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:44:15 +00:00
d9bc36cde3
chore(scheduler-sim-discrete-clock): HA soak with the unmodified --sim-speed=0 caller passes (task 5.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:39:47 +00:00
69e2f07811
chore(scheduler-sim-discrete-clock): full ASAN regression green, 117/117 (task 5.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:31:28 +00:00
d5527c665b
docs(scheduler-sim-discrete-clock): scheduler-core spec drops speed modes, --sim-speed deprecated, REPL speed removed (task 4.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:26:03 +00:00
66435d100d
docs(scheduler-sim-discrete-clock): deployment guide describes the event-driven sim clock and deprecated --sim-speed (task 4.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:21:11 +00:00
aa2bb6d090
feat(scheduler-sim-discrete-clock): remove the simulator speed argument and shadow copy (task 3.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:19:34 +00:00
eb5f339541
feat(scheduler-sim-discrete-clock): retire --sim-speed as a warn-once no-op, end-to-end CLI tests (task 2.3)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:06:06 +00:00
05320e777b
feat(scheduler-sim-discrete-clock): remove the REPL speed parameter (task 2.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 04:53:20 +00:00
3a6b503f07
feat(scheduler-sim-discrete-clock): pin the simulation clock so it moves only at recorded timestamps (task 2.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 04:41:14 +00:00
4f653d8fbe
feat(scheduler-sim-discrete-clock): --sim-speed deprecation-warning helper and discrete-clock test binary (task 1.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 04:23:46 +00:00
4b6872d053
docs(spec): scheduler-sim-discrete-clock requirements, design and tasks approved
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 04:16:17 +00:00
dcf3acf32a
feat(libxce-alloc-preason-equality): full ASAN regression green (task 5.2); 5.3 blocked on scheduler-sim-discrete-clock
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 11:11:41 +00:00
a9c984775f
feat(libxce-alloc-preason-equality): drop sim verify tests that duplicate libxce alloc_eq coverage (task 5.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 11:04:19 +00:00
24b057189f
feat(libxce-alloc-preason-equality): sim verify uses public job_alloc_eq and logs the first difference (task 4.3)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:58:41 +00:00
69112c2d22
feat(libxce-alloc-preason-equality): recorded dispatch fixtures carry the session-start from (task 4.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:44:00 +00:00
6110d12e5a
feat(libxce-alloc-preason-equality): white-box equivalence tests use public job_alloc_eq/job_preason_eq (task 4.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:27:46 +00:00
c1960457f8
feat(libxce-alloc-preason-equality): stamp every allocation's from with the deciding session start (task 3.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:14:35 +00:00
2d7ba66019
feat(libxce-alloc-preason-equality): record session begin from the session-start clock reading (task 3.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:59:56 +00:00
62d0b8b686
feat(libxce-alloc-preason-equality): full-match job_preason_eq with first-difference description (task 2.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:49:57 +00:00
a6cbb323b0
feat(libxce-alloc-preason-equality): public alloc_eq/host_alloc_eq/job_alloc_eq with first-difference description (task 2.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:35:08 +00:00
8f25c1c3f3
feat(libxce-alloc-preason-equality): NULL-safe str_eq helper in libxce utils (task 1.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:21:04 +00:00
094577f245
docs(spec): libxce-alloc-preason-equality tasks approved for implementation
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:13:29 +00:00
dcf54387ae
docs(spec): libxce-alloc-preason-equality review fixes; new scheduler-sim-discrete-clock spec
libxce-alloc-preason-equality (requirements, design, tasks; not yet approved):
- alloc_eq compares `from` and excludes only instid; the scheduler stamps
  `from` with the deciding session's start, the same clock reading that
  is recorded as beginRound, so a replay reproduces it.
- every comparator takes an optional char** why: a malloc'd description
  of the first difference, owned and freed by the caller; --sim-verify
  mismatch lines print it (resolves bug-1881).
- job_preason_eq is new full-match behavior, not a relocation (bug-1845
  keeps the simulator main-reason-only); the three white-box test files
  switch to it and keep their sim_verify.c include.
- alloc_eq stays separate from merge_job_alloc()'s same_alloc_slot()
  (bug-1863), which does match on instid.

scheduler-sim-discrete-clock (initialized): remove --sim-speed so the
simulator clock moves only at recorded timestamps and a replayed
session_start equals the recorded beginRound exactly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 08:50:07 +00:00
aa7fc65a36
fix(controller): cold start elects in ~1.7 s, not ~10 s; ctrlstatus appliedLsn on the primary (bug-1879, bug-1878)
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
  follower with no leader known; the liveness poll then retries the
  election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
  heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
  then only holds the watchdog's arm time, and the failover gates took
  it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).

bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.

Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 07:35:45 +00:00
b63bb581d4
chore(buglog): point fix references at the re-signed commit hashes
The 16 commits since 534895c5 were re-signed (rebase --exec amend -S; tree
unchanged), which changed their hashes. Update the buglog and the
scheduler-allocation completion note to the new ones.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 13:15:43 +00:00
4ba0f32766
chore(scheduler-allocation): mark implementation complete
All 16 subtasks were already done; check the 6 parent tasks and set
phase implementation-complete. Verified 2026-09-24/25: scheduler suite
87/87 (ASAN), live multi-host e2e PASS, HA soak sequential + chaotic PASS.
Out-of-scope follow-ups stay open: bug-1766, bug-1768, bug-1769,
bug-1771, bug-1860.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 13:15:03 +00:00
b0f2aea466
chore(gitea-sync): record first status-driven Gitea sync
- buglog: gitea_issue / gitea_synced_status for the 20 bugs created and the
  30 already-synced entries (#25/#35 reopened as partial)
- .kiro/specs/*/spec.json: "gitea" milestone/issue mapping for all 66 specs
  (65 milestones + 195 issues created, 158 closed at creation)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 10:22:05 +00:00
6e0463e707
chore: persist OpenWolf state and libxce-alloc-preason-equality spec
- .wolf/anatomy*, memory.md: hook-maintained index and session log
  through the bug-1851..1854 work.
- .kiro/specs/libxce-alloc-preason-equality/: requirements, design,
  research and tasks generated; none approved yet (spec.json phase
  tasks-generated).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 08:44:44 +00:00
668e3f6893
chore: persist OpenWolf state, controller-peers-env-fallback spec, and scheduler backlog notes
- .wolf/anatomy-index.json, anatomy.md, memory.md: OpenWolf hook-maintained
  project index and session log, refreshed by this session's activity.
- .kiro/specs/controller-peers-env-fallback/: spec scaffold (brief,
  requirements, design, research) from prior spec-init work.
- issues/new_requirements.md: scheduler simulator enhancement backlog
  (info-command improvements, pend-reason detail, decision tracing).

Excluded as non-persistent: issues/Issues.md (empty placeholder),
src/lib/testsuite/test_xcectx_reconnect_race.c (self-documented as an
ASAN stress reproduction only, not a committed regression test, and not
wired into meson.build), scheduler-replay-20260922-095207.replay
(4MB binary debug capture, no .replay files are tracked in this repo).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 03:19:52 +00:00
4f94638a1d
fix(scheduler): route sim reconciliation's on_dispatch() through the allocator thread (bug-1837)
sim_reconcile_unaccounted_jobs() (sim_loop.c) was calling
picker->picker_iface->on_dispatch() directly from the sim driver thread,
violating plugin.h's documented "allocator thread only" contract for
Post-allocation hooks. Dormant in practice only because builtin_fcfs.c
happens to lock internally around every hook -- a defensive measure the
contract text doesn't actually require of picker plugins.

Since scheduler-sim-live-allocation, sim mode spawns the same real
sched_job_picker()/sched_allocator() thread pair normal mode uses, so a
real allocator thread now exists to route through. Added a
reconcile_req/reconcile_ack cbuf pair on xce_sched_core_t and a reserved
XCE_SCHED_RECONCILE_WAKE sentinel pushed through jbuf to wake the
allocator thread without touching jbuf's pinned capacity-1/item-type
invariant (Req 5 AC 7). sched_allocator() services the request and calls
on_dispatch() itself; sim_reconcile_invoke_on_dispatch() blocks on the ack.
Falls back to a direct call when no allocator thread is live for a given
core (standalone unit tests), which is safe by construction.

TDD: new test spawns a stand-in allocator-relay thread and asserts
on_dispatch() fires there, not on the caller's thread -- confirmed RED
against the old direct-call code, GREEN after. Four other test files with
hand-rolled sched_allocator() stubs needed the same relay branch (found by
an actual hang at meson's timeout). Full scheduler suite: 85/85, zero
regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-21 12:58:57 +00:00
94115b0024
fix(scheduler): sim mode force-dispatched recorded decisions past live capacity (bug-1836, bug-1838)
User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:

- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
  Reconciliation" backstop) force-applied any recorded DISPATCH decision
  it found, with zero live capacity check. Added
  sim_reconciled_alloc_fits_capacity(), reusing the same
  sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
  the live picker/allocator path already uses.

- bug-1838: even with that gate, reconciliation still preempted every
  freshly-PEND job before the live picker/allocator got a turn each round
  -- unnecessary now that scheduler-sim-live-allocation gave sim mode the
  same real, capacity-checked picker/allocator threads normal mode uses.
  Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
  2026-09-11 widening whose own justification no longer holds), so a
  freshly-pushed job now always flows through the same live, verified
  decision path as normal mode. Reconciliation now only ever fires for a
  job already RUN from before the replay's own observation window --
  the one case structurally invisible to the live picker.

Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.

Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.

Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-21 09:50:27 +00:00
bf61f5954e
test(scheduler-sim-diagnostics): close cross-cutting gaps for Requirement 6 (task 10.2)
Final validation task for the Human-Readable Units increment (tasks
7-10.1). Adds three tests closing gaps no prior task's own tests
covered:

- Cross-surface consistency (Requirement 6.5): a single info limit
  request against a byte-valued limit with both a configured setting
  and a live usage instance shows the SAME units convention in both
  lines, across mode switches -- neither task 8.2's nor task 9's own
  tests combined both surfaces in one capture.
- Non-byte-valued exclusion (6.3): a new "slots"-resourced fixture
  (the first non-"mem" fixture in this file) proves the entire
  combined display is byte-for-byte identical across all three units
  modes.
- Percentage exclusion (6.3) at the REPL-dispatch level: a percent
  limit's own percentage value survives real repl_dispatch()/
  set-units switching unchanged, complementing test_limit.c's
  existing library-level proof of the same invariant.

Full regression sweep: lib suite 368/368, scheduler suite 85/85,
including test_repl_limit_table.c (35/35), test_repl_resource_table.c,
test_repl_resource_formatting.c, and test_limit_plugin.c all
unaffected. No production code changed. This closes out Requirement 6
(ACs 6.1-6.5) and reconfirms Requirement 5 (ACs 5.1-5.3) end-to-end.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-20 08:08:21 +00:00
5125536657
test(scheduler-sim-diagnostics): strengthen coincidentally-passing assertions (task 10.1)
Six pre-existing assertions in test_repl_limit_table.c (across 4 test
functions) used bare strstr() substring checks against whole-number
fixture values on "mem" -- e.g. strstr(captured, "accounted=4.00").
Task 9's fix changed the actual output to "accounted=4.000000 (4.00
B)", but the old substring is a literal prefix of the new one, so
these assertions kept passing by coincidence of formatting, providing
zero actual regression protection for the fix.

Strengthened each to assert the full corrected "<raw> (<human>)"
string. Proved via TDD: reverting task 9's fix now makes all 6
strengthened assertions (plus task 9's own 2 genuine tests) fail;
restoring the fix returns the suite to 85/85 green.

Confirmed via full-file scan that no other loose assertion of this
shape remains, and that info worker/info resource's own regression
suites were never vulnerable to this class of bug (they build expected
strings dynamically rather than hardcoding literal substrings).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-20 07:35:11 +00:00
701901079e
fix(scheduler-sim-diagnostics): correct usage display's resource classification (task 9)
repl_print_limit_usage()/repl_print_limit_usage_per_host() (repl_info.c)
passed vres->vres->name (a synthesized compound scope-instance name, e.g.
"limits2/u=rshen/q=xce/p=") to the byte-value classifier instead of the
limit's actual constrained resource name -- so a byte-valued limit's
usage amounts never got recognized as byte-valued, regardless of the
underlying formatting machinery already working correctly.

Fixed both call sites to pass the limit's own resname
(limit->val.resname / vres->limit->val.resname) instead. Added tests
proving human-readable rendering for both the cluster-wide/GLOBAL path
and a percentage-based per-host limit (Requirement 6 AC 6.4: a
percentage-of-byte-resource limit's resolved absolute amount is
human-formatted too).

Discovered and confirmed (implementer + independent reviewer): the
fix landed with zero pre-existing test failures, not the handful
design.md predicted -- those assertions used whole-number fixture
values with bare strstr() substring checks, which still match the
new, longer human-augmented output by coincidence of formatting.
Documented in tasks.md; task 10.1 will strengthen those assertions
to genuinely exercise the fix rather than pass by accident.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-20 06:39:49 +00:00
8477f677e4
feat(scheduler-sim-diagnostics): wire info limit to fmt_limit_units (task 8.2)
Switches both fmt_limit() call sites in repl_info_limit() (the "all"
path and the named-lookup path) to
fmt_limit_units(..., translate_units_mode(repl_current_units_mode(sim))),
so a byte-valued limit's configured amount now renders human-readable
under the REPL's real, currently-selected units mode -- the same
convention info worker/info resource already honor.

Adds two tests proving human-readable rendering under the default
mode and consistent switching across set units raw|human|both via
real repl_dispatch(). RED confirmed by reverting the call sites
(2 failures, 28 pre-existing tests in the same file unaffected).

Corrects an inaccurate self-reported regression-safety rationale
(there is no slots-resource fixture in this file; safety instead
comes from those tests never asserting on the formatted value line),
logged in tasks.md's Implementation Notes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-20 05:58:41 +00:00
9f54c5753f
feat(scheduler-sim-diagnostics): add fmt_limit_units() sibling (task 8.1)
Adds fmt_limit_units(buf, len, limit, units_mode) to src/lib/xcelimit.c,
a units-mode-aware sibling of fmt_limit() following the
fmt_jobid()/fmt_jobid_ex() extended-sibling pattern. Reuses fmt_limit()'s
exact consumer-dump loop untouched; only the final value line branches:
percent limits always render bare "%f%%" regardless of mode, byte-valued
(or "swp"-special-cased) absolute values render via
xce_format_units_value() (task 7), everything else keeps the
unconditional "%f".

fmt_limit() itself is now a one-line XCE_UNITS_RAW wrapper around
fmt_limit_units() -- provably byte-identical to its prior output for
every input shape, so its 5 existing direct unit tests needed zero
changes (the regression proof this refactor relies on).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-20 05:09:09 +00:00
eb23c158ae
feat(scheduler-sim-diagnostics): promote byte-unit formatting to shared lib (task 7)
Adds Requirement 6 (Human-Readable Units for Byte-Valued Limit Amounts)
to the spec: requirements, design, research, and tasks 7-10.

Task 7 (Foundation): promotes the REPL's byte-unit scaling algorithm
and byte-valued-resource classifier into src/lib/utils.c/utils.h
(xce_format_bytes_human, xce_is_bytes_valued_resource), alongside a
new display-mode-aware composer (xce_format_units_value,
xce_units_mode_t) -- following the fmt_jobid()/fmt_jobid_ex()
precedent. repl_fmt.c's scheduler-only functions become one-line
delegating wrappers. repl_info.c's repl_format_numeric_resource_value()
(shared by info worker/info resource, not just the upcoming info
limit) is retyped to delegate to the new shared composer via a small
translate_units_mode() bridge, so every consumer benefits from one
implementation going forward.

No existing symbol's signature changed; only new symbols were added.
Confirmed behavior-preserving: full scheduler suite 85/85 unchanged,
including info worker/info resource's own regression tests. One
narrow, untested, out-of-requirement-scope corner case found and
documented (swp's RAW/BOTH mode rendering under the new single-value
composer interface) -- logged in tasks.md's Implementation Notes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-20 04:40:44 +00:00
984a686974
docs(scheduler): correct stale bug-1611 status and driver-equivalence steering
The dimension-1 audit (previous commit) reported bug-1611 as only
"partially fixed" -- but that assessment worked from bug-1611's own
stale narrative text without checking scheduler-sim-eager-reconciliation
(a separate spec, completed 2026-09-11, predating this session's audit).
That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger
from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in
the same round it is recorded, never deferred to a later confirmation
event -- closing bug-1611's original motivating scenario for good.

The "residual gap" my own first correction pass invented (an
already-RUN-at-snapshot-load job) is also not real: such a job's
allocation comes from the snapshot directly (confirmed field-complete
by this session's dimension-4 audit), so it's never a reconciliation
candidate in the first place.

Corrected bug-1611's buglog entry to FIXED and updated
scheduler-driver-equivalence.md's stale "Known violations" section
(still described both bug-1611 and bug-1613 as open) to reflect current
reality, with a pointer to re-verify rather than re-trust either framing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-19 14:01:09 +00:00
2f99406788
docs(scheduler-sim-live-allocation): validate against real recorded sessions (task 5.4)
Deployed a real controller+scheduler+worker cluster and used the real
xceadmin scheduler record start/stop production path (no synthetic
corpus existed in this checkout -- bug-1611/1734's referenced replays
were stale /tmp paths, long gone) to capture two genuinely
live-executed .replay sessions (110 rounds/3 jobs, 28 rounds/5 jobs).

Ran --sim-verify against both: zero MISMATCH in either. Session 1
organically exercised the real deployment's global limit1 (cap=3),
correctly PENDing two 22-task jobs for the whole session -- root-caused
live via resource.c's own DEBUG3 trace and confirmed as pre-existing,
driver-agnostic enforcement (identical in normal mode), not a defect.
Session 2 cycled 15 PEND + 5 DISPATCH decisions under the same limit,
directly exercising the live-decision-authoritative rule under real
multi-round timing.

This closes out all 14 leaf tasks (1.1-5.4) of the
scheduler-sim-live-allocation feature.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-19 10:32:50 +00:00
aacb7250e3
docs(scheduler-sim-live-allocation): reconcile equivalence suites (task 5.3)
Re-ran test_repl_equivalence.c, test_sim_equivalence.c, and
test_sim_contention_parity.c in full: all pass, 85/85, zero assertion
changes anywhere. Confirmed all three still exercise their own
link-satisfying always-PEND sched_job_picker/sched_allocator stub
(added task 4.1), correctly proving the new two-thread jbuf timing
didn't disturb decision ordering -- but not proving real live-path
resource-based dispatch parity, which is already covered separately
by tasks 4.1/4.3/5.1/5.2's real-scheduler_srcs-linked fixtures.

Fixed a stale documentation defect: all three files' header/inline
comments claimed sim mode "structurally CANNOT emit a DISPATCH
decision" via sim_picker_thread_fn(), a function this feature removed
in tasks 3.1/4.1. That claim is now false of production sim mode
(disproved by test_sim_live_allocation.c and others); corrected with
a RECONCILIATION addendum rather than rewriting the original
historical rationale. No production code or assertion values changed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-19 09:39:32 +00:00
a70c7a2c61
test(scheduler-sim-live-allocation): prove replay-driven limit visibility (task 5.2, closes bug-1826)
Adds the feature's concrete original acceptance test: a genuine
capnp-encoded .replay (real ReplayHeader/SnapshotEntry/Conf4Sched with
one global and one host-scoped limit), driven end-to-end through the
real, unmodified sched_sim_driver_run(). The dispatched job's usage
registers against both limits via limit_get_configured_limits()/
limit_get_usage_instances() -- the same accessors info limit reads --
proving real, non-zero usage where today's REPL previously showed none.

RED-phase evidence: reverting task 4.3's 4 files to their pre-fix
commit reproduces the exact predicted failure (host-scoped usage
missing, global usage intact); restoring to HEAD reproduces GREEN.

Closes bug-1826 (the bug this whole feature was built to fix) --
buglog entry updated from NOT FIXED to FULLY FIXED, citing this test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-19 09:11:40 +00:00
ab88848f91
test(scheduler-sim-live-allocation): prove contention parity across drivers (task 5.1)
Adds two new fixtures spawning the real sched_job_picker/sched_allocator
thread pair directly against bare normal-mode and sim-mode core state
under genuine resource contention (2.0 capacity, 3 jobs x 1.0 demand),
asserting identical admission decisions and identical accounting state
(job->accounts, w->resInUse), with and without a matching host-scoped
limit. The pre-existing test_sim_contention_parity.c could not be reused
since its white-box stubs hard-code the old always-PEND behavior and
never exercise live election/allocation.

Confirms build_limit_vres_requests()/get_vres_value() (plugins/limit.c)
are #if 0'd dead code with zero live callers -- no limit in this codebase
currently gates/denies admission in either mode, only tracks/displays
usage -- so this proves usage-tracking parity, documented explicitly as
an honest scope boundary rather than a silent narrowing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-19 08:24:35 +00:00
f6a2862110
feat(scheduler-sim-live-allocation): close host-scoped limit fan-out seam gap (task 4.3)
Fully closes bug-1829's own tracked follow-up from task 4.1: retypes
seqjob_allocator.c/parjob_allocator.c's init_fn AND runtime hooks
(prepare_allocation_hook, job_alloc_fn) to read the plugin's own
honest plugin->core field (task 1.2) instead of deriving an
xce_scheduler_t* from plugin->sched -- mirroring task 2.3's limit.c
migration exactly. The gap was broader than initially scoped: not
just init_fn, but every runtime hook independently derived sched from
plugin->sched on each allocation attempt.

Adds seqjob_allocator_new_plugin()/parjob_allocator_new_plugin()
factories (plugin.h, mirroring limit_new_plugin()), letting
sim_alloc.c's bootstrap functions register both plugins DIRECTLY
against &(sim->core) via the existing plugin_register_core() -- the
same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already
use. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the
plugin-registry/fan-out path entirely; plugin_prepare_allocation()'s
fan-out during a live seqjob/parjob dispatch now genuinely reaches
sim->core.plugins, the same registry the limit plugin is registered
in.

One narrow, necessary exception remains, independently verified
confined: my_alloc() still needs a small xce_scheduler_t wrapper
(only .core.resmgr aliased to sim->core) to satisfy sched_make_allocs()
(scheduler.c, out of this task's boundary, unmodified) -- set on
plugin->sched only AFTER plugin_register_core() has run, never read by
the plugin registry or any prepare_allocation_hook fan-out.

TDD: new test_sim_alloc_limit_fanout.c drives a real live dispatch
(sched_job_picker()/sched_allocator() thread pair) with a host-scoped
limit registered directly against sim->core, proving a real usage
instance is created -- isolated from both reconciliation and
global-limit matching. RED: dispatch succeeded but the usage-instance
iterator was empty. GREEN: 82/82, zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-19 07:38:34 +00:00
915782fc66
feat(scheduler-sim-live-allocation): narrow reconciliation to the timing-gap safety net (task 4.2)
Adds a plugin_job_election(&(sim->core), job) call to
sim_reconcile_unaccounted_jobs()'s FOUND branch, before the existing
alloc-reconstruction/state-transition code, so a job whose outcome
comes from the recording still gets its global-limit usage registered
-- the same core-safe dispatcher the live path (task 4.1) already
uses, not a separate mechanism.

The existing job->accounts != NULL trigger guard needs no change: it
already excludes any job the live path has already decided, so this
call is naturally idempotent (match_job_global_limit_'s own
obj->configver guard makes a redundant call a safe no-op) -- narrowing
this function to a genuine timing-gap safety net rather than the
general-case decision source it used to be, per design's Live/Recorded
Authority Rule.

Host-scoped limit matching is deliberately not attempted here (needs a
resolved worker context this function doesn't independently
construct) -- that gap is tracked separately as task 4.3.

TDD: new tests prove a reconciled job's global limit is genuinely
registered via the real limit_get_usage_instances() accessor, and that
a job the live path already decided is never reprocessed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-19 05:57:55 +00:00