The 16 commits since 534895c5 were re-signed (rebase --exec amend -S; tree
unchanged), which changed their hashes. Update the buglog and the
scheduler-allocation completion note to the new ones.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since bug-1790's fix (9caf1424) counts a job's full task total against
global limits, the repo's limit1 (all users except rshen share 3 slots)
pends xce_test's root-submitted 33-task job forever, so the multihost
e2e fixture failed even though the allocator is correct.
deploy-ha.sh gains an optional XCE_DEPLOY_CONFIG_DIR (default unchanged:
src/configs). run-multihost-verify.sh copies src/configs into its deploy
dir, replaces limits.yaml with an empty list, and deploys with that
override.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- buglog: gitea_issue / gitea_synced_status for the 20 bugs created and the
30 already-synced entries (#25/#35 reopened as partial)
- .kiro/specs/*/spec.json: "gitea" milestone/issue mapping for all 66 specs
(65 milestones + 195 issues created, 158 closed at creation)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gitea-sync.py fixes found on the first live run (bug-1855 follow-up):
- 204/empty-body replies (label DELETE) no longer crash in json.loads
- every request has a 60s timeout, so a stalled call fails instead of hanging
- spec.json sync state is inserted as a trailing "gitea" block instead of
re-serializing the file (15 hand-formatted spec.json files would reflow)
- implementation issue body labels generated vs approved correctly; the
validation body uses the same done-phase set as the close rules
Project memory now routes all buglog writes through the buglog skill:
CLAUDE.md key rule, a cerebrum Do-Not-Repeat entry (with the older
duplicate-id / json.dump / tags-vs-fix lessons tagged as superseded), and
xce-audit-invariants step 5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog had no status field, so "is this bug open?" meant reading free
text, and gitea-sync closed any issue whose `fix` text was non-empty --
including "NOT FIXED"/"PENDING"/"PARTIALLY FIXED" ones (bug-1855).
- Backfill `status` (+ status_updated) on all 1,193 entries. The 85 entries
whose text said not-fixed/pending/out-of-scope were checked against HEAD
and git history; the rest are `fixed`. Result: 36 open, 10 partial,
2 investigating, 5 cannot-reproduce, 4 wontfix, 1 duplicate.
Insert-only diff: existing lines are untouched.
- .claude/rules/buglog-status.md defines the status values (kept out of
openwolf.md, which `openwolf update` overwrites).
- scripts/buglog.py + buglog skill: add (next free id), status, note,
recur, list, show, triage. Edits only the touched entry's lines; a full
json.dumps rewrite churns ~1,200 lines of mixed \u-escaping.
- gitea-sync (script + skill): state and a single bug:<status> label come
from `status`; reopen supported; issues shared by several entries follow
the most-open one; only status labels are swapped; per-entry write-back
with gitea_synced_status so unchanged entries cost no API calls; list-typed
`fix` no longer crashes; spec issues close on done phases (spec.json has
no implementation approval); --since no longer floods Gitea after the
backfill.
- Disable OpenWolf buglog auto-detection: its hook numbers new entries
len(bugs)+1, which collides with existing ids (bug-1856, upstream).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- .wolf/anatomy*, memory.md: hook-maintained index and session log
through the bug-1851..1854 work.
- .kiro/specs/libxce-alloc-preason-equality/: requirements, design,
research and tasks generated; none approved yet (spec.json phase
tasks-generated).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A chaotic round's walcheck compared all three members' on-disk WALs with
boundary=auto. The just-killed term-4 primary still held its NACKed,
never-committed lsn=58, which differed from the new term-5 leader's
own-term noop at lsn=58, and was counted as class=committed.
walcheck_count now takes an optional "midloop" mode, used only by the
per-round chaotic check: the boundary is the serving primary's own
committedLsn (live_committed_boundary() needs 3-of-3 convergence, which a
chaotic round never reaches), and only members guaranteed to hold the
committed prefix are compared -- the primary plus live standbys whose
appliedLsn has reached the boundary. Every other caller, including the
settled final sweep, still compares all three. Log lines name members=.
Also logs bug-1854 (scheduler re-adds every configured resource on each
rejoin config push; ERROR noise, changed definitions ignored) for
follow-up.
10-round chaotic soak: PASS (divergence 0, job loss 0, SIMVERIFY 9 files,
0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The recorder encoded each controller event batch after
xce_core_push_events() had applied it, and applying moves payload fields
out of the events: a WORKERS_PUSH's resvals (core_apply_worker_resvals
MOVE_PTR), a JOBS_PUSH's allocs, a JOB_RUSAGE's rusages, and a
CONFIG_PUSH's shared resvals (sched_apply_pending_config recorded after
sched_init_config). Recordings silently lost exactly those fields -- a
worker's resval names were written with NULL values, so --sim-verify
replayed "no slots" and turned 3 live DISPATCHes into PENDs.
driver.c now records the batch before the push; the periodic in-stream
SnapshotEntry moves out of sched_record_write_events() into new
sched_record_checkpoint_after_apply(), called after the push so a
checkpoint still reflects the events written ahead of it. The config
push is likewise recorded before sched_init_config().
Regression test: test_recorded_workers_push_keeps_resval_values (RED:
recorded value NULL; GREEN after). Periodic-checkpoint tests updated to
the write-then-checkpoint contract. meson test 109/109; 25 live soak
recordings replay with 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_val_res() memcpy'd the xce_val_t into w->host.resvals, so the
"type"/"model" string resources aliased the interned xstrings owned by
w->host.attrs. rejoin_worker() drops the old attrs strings, then
apply_join_state() -> resync_arm_round() resolves the previous join's
still-outstanding round, firing the worker hooks before
add_hostattrs_resvals() re-seeds resvals; the apiserver hook's
collect_resval() strdup()ed the freed hostmodel. Live: a newly promoted
primary died ~8s after election during an HA soak.
A string resval now stores a private strdup (taken before the old copy is
freed), and the owning arrays free their entries: host.resvals gets
kill_resval (replacing the FIXME NULL killer) and c->sharedinst's
resval_obj_killer (previously a no-op) frees the string and the value --
both also leaked every xce_val_t before. Also closes the same alias for
worker-reported string resvals and config values.
Regression test: test_rejoin_worker_string_resvals_survive_outstanding_round
(RED reproduced the live ASAN stack; GREEN after). meson test 109/109.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follow-ups from the bug-1846 investigation.
bug-1847: config pushes never reached a .replay recording -- handler.c
applied them synchronously, bypassing the pending-update queue the
recorder writes from. With bug-1846's deferral they're now applied at a
round boundary on the driver thread, so sched_apply_pending_config()
writes a CONFIG_PUSH event via the existing sched_record_write_events()
(no-op when not recording), ahead of the round's other recorded events.
Test: test_config_push_deferred.c's test_applied_config_push_is_recorded
(RED: 0 events; GREEN: 1). Fidelity limit, pre-existing: replay applies
configPush through core_events.c's CONFIG_PUSH path, which doesn't re-run
groupmgr_load_configs()/plugin_config_update(), so a limits/groups change
still won't take effect in sim.
bug-1848: xce_sched_obj_t.hlimits was created with a NULL zarray killer
at all three constructors, so reset_sched_obj() and schedobj teardown
leaked the per-worker xlist_t match_job_host_limit_() callocs. New
kill_hlimits_list() (schedobj.c) frees the list, never its vres entries
(owned by the limit plugin); core_events.c and snapshot_load.c use
identical static local copies, since both avoid sharing helpers with
schedobj.c and several test binaries link them without it. Test:
test_limit_host_dedup.c's test_hlimits_zarray_releases_its_lists (RED:
killer NULL; GREEN).
bug-1849: every --sim run warned "failed to resolve job picker plugin ...
picker rebuild will be skipped". Benign: the snapshot is loaded before
the sim registers its picker, and sim_bootstrap_picker() replays the same
queue/job walk afterward. Downgraded to TRACE with an explanation.
Buglog also records bug-1850 (open): the ~100s reconnect churn behind
bug-1846 part 2 -- xcecli_retry_next() rotates/reconnects on every retry
(ETIMEDOUT after the 10s reply deadline, E_XCE_NOT_READY, redirects), so
a slow reply from a healthy primary forces a rejoin + full re-push.
Needs the live scheduler.log and a libxce spec.
Full scheduler suite 87/87 (asan). Replay of the captured file: same 184
mismatches, all attributable to the recording's pre-fix binary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.
Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.
Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.
Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
-- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
returned NULL right after the push), GREEN (session walks the remaining
jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
#include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).
Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
match_job_host_limit_() (plugins/limit.c) runs on every per-host
allocation attempt via limit_prepare_allocation(), and unconditionally
xlist_store()d the matched vres into obj->hlimits[worker->idx]. That list
is only rebuilt by limit_job_election() on a configver bump, so a job that
kept pending on the same host gained one more copy per attempt.
sched_fill_limit_requests() then staged every copy into
ctx->alloc_req.host, and seqjob's make_host_alloc() built a host alloc
carrying all of them.
Found via temporary per-decision --sim-verify diagnostics (reverted, not
committed) against scheduler-replay-20260923-085356.replay: pc13v host
allocs held mem + slots + ~70 identical limit entries (72) instead of 3.
After the fix the replay side is always 3 entries, and the LIVE recording
itself still shows the inflation (72/81/96/114/138/177 entries, resetting
only on a config push) -- so this is a production bug, not sim-only.
make_job_account() creates one res_account per entry and
sched_update_res_accounts() charges each, so the host limit was charged
(and later released) N-fold.
Fix: hlimits_contains_() pointer-identity check (vres is interned per name
in privdata->instances); a matched vres is stored once per (job, worker).
Test: test_limit_host_dedup.c (sched_limit_host_dedup_test) -- five
plugin_prepare_allocation() attempts for one job on one worker must leave
hlimits[worker]->count == 1 and one staged copy of the vres in
alloc_req.host. RED before the fix (Expected 1 Was 2), GREEN after; full
scheduler suite 86/86 (asan).
Still open (bug-1846 part 2, logged in buglog): 9 live-recorded rounds end
after 11-12 decisions while replay processes the whole candidate list; not
root-caused, needs the live scheduler.log or a fresh capture.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/xce-audit-invariants scheduler dimension-1/4 audit, prompted by a fresh
user-captured .replay file that still showed massive --sim-verify mismatch
volume after bug-1843's fix. Root-caused via temporary diagnostic
instrumentation against the real capture (reverted, never committed) rather
than trusting the surface symptom.
bug-1844: sim_verify_alloc_eq() (sim_verify.c) compared `instid`, a bare
per-process monotonic counter (scheduler.c's sched_get_id(), core->nextid)
stamped at allocation-build time -- not semantic content. The original
recording's counter and a fresh replay's counter have no reason to reach
the same value for "the same" dispatch. Live-verified: every DISPATCH
mismatch showed identical value/duration/resname/reserve, differing only
in instid (e.g. 826 vs 1). Dropped from the comparison, same treatment as
bug-1842's `from` exclusion.
bug-1845: sim_verify_job_preason_eq() required PEND decisions' full
host/resource reason-detail lists to match. core.h documents "simulation
isolation" as deliberate design: sim/replay never registers
core->unsatisfiable_lookup_cb, so it can never hit the unsatisfiable-cache
main.c's normal-mode branch registers. A live cache hit
(sched_pend_from_cached_reason_()) records only main_reason with zero
detail; replay always falls through to the full per-host computation and
always produces detail. Same semantic outcome (PEND, same main_reason) at
two architecturally-legitimate detail levels. Relaxed the comparator to
main_reason only. Separately verified the cache itself is sound (key is
safe within a round since capacity only decreases and the cache clears
every round) -- ruling it out as the cause of a second, still-open
divergence (bug-1846: one job genuinely dispatches one round early in
replay vs. the original recording -- a suspected resource-accounting
parity gap, not yet root-caused, logged for follow-up).
bug-1772 (previously deferred, incorrectly tagged "fixed"): re-found
independently by a dimension-4 snapshot-field-completeness audit subagent.
snapshot_build_primary_schedobj() (snapshot_load.c) never computed
resreq_fp, unlike its declared model core_build_primary_schedobj()
(core_events.c) -- every snapshot-restored job got a zeroed fingerprint,
which alloc_cache.c's cache key folds in directly alongside
user/queue/project, so two distinct restored jobs sharing those three
fields could silently share an incorrect unsatisfiable-cache verdict.
Added the same 4-field computation the live path uses.
TDD throughout: each fix has a dedicated regression test, and each was
verified via genuine red (revert fix, confirm the new test fails) then
green (restore, full 85/85 scheduler suite). Also ran three parallel
dimension audits (core I/O isolation, picker/allocator jbuf rendezvous) --
both clean, no violations found.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sched_record_session_add_decision() (recording_session.c) grows its
per-round decisions array via realloc(), doubling from 8. On a failed
growth it silently dropped the current decision and every one after it
that round, without marking the round incomplete -- finalize() then
wrote the short array as an ordinary, apparently-complete ReplaySession.
--sim-verify's replay comparison had no way to distinguish that from a
genuinely smaller live round, producing spurious "expected: <none>"
mismatches for jobs that were, in reality, correctly decided live and
simply never recorded. Live scheduling itself was unaffected either
way -- only the .replay file's own record of that one round lost its
tail.
This refutes the leading theory from the prior investigation (replay
seeing a job's first appearance "one round early" via coarser event
batching in sim_loop.c) -- confirmed via targeted diagnostic
instrumentation against the original captured .replay file: replay
deterministically reproduces the full candidate set every round;
the recording itself has the one-round gap.
Fix: track truncation on the session ctx; finalize() now refuses to
write an incomplete session and fails loudly through the same
sched_record_abort_on_error() path every other recording failure in
this subsystem already uses.
TDD: added a regression test forcing a real, deterministic realloc()
failure via -Wl,--wrap=realloc (scoped to this one test binary --
no prior malloc/realloc fault-injection harness existed in this
codebase). Verified genuine red -> green: the test fails without the
fix, passes with it. Full 85/85 scheduler suite green.
Note: this prevents future recordings from losing decisions this way;
it cannot retroactively repair an already-captured .replay file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- .wolf/anatomy-index.json, anatomy.md, memory.md: OpenWolf hook-maintained
project index and session log, refreshed by this session's activity.
- .kiro/specs/controller-peers-env-fallback/: spec scaffold (brief,
requirements, design, research) from prior spec-init work.
- issues/new_requirements.md: scheduler simulator enhancement backlog
(info-command improvements, pend-reason detail, decision tracing).
Excluded as non-persistent: issues/Issues.md (empty placeholder),
src/lib/testsuite/test_xcectx_reconnect_race.c (self-documented as an
ASAN stress reproduction only, not a committed regression test, and not
wired into meson.build), scheduler-replay-20260922-095207.replay
(4MB binary debug capture, no .replay files are tracked in this repo).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sim_verify_alloc_eq() compared xce_alloc_t.from, which make_alloc()
(xcealloc.c) stamps with the real OS wall clock (time(NULL)), not
core.c's virtualized sim clock. The recorded (expected) value is
whatever real second the original run happened to execute in; the
freshly-rebuilt (actual) value is whatever real second the replay
executes in -- two unrelated real-world timestamps, so every
otherwise-correct DISPATCH decision failed verification whenever a
replay ran even one wall-clock second away from its own recording.
Drop `from` from the comparison -- it's reservation bookkeeping, not
part of a decision's semantic identity. Verified against a real
captured .replay file: 120/120 false DISPATCH mismatches -> 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker-rejoin-resync sets XCE_WORKER_RECONCILING and pushes it to the
scheduler, which correctly freezes that worker's resInUse mid-reconciliation
(bug-1799/1801). But resync_resolve_round() only cleared the bit on the
controller's own worker_t, never notifying the scheduler -- so once a
rejoin round resolved, the scheduler's copy stayed RECONCILING forever,
permanently freezing that worker's resource accounting even after every
job on it finished. Live-observed via a replay capture: a 22-slot worker
stuck at used=22.00 for the rest of the run with zero RUN jobs left.
Added persist_notify_worker_status_updated() (mirrors bug-1679's job-side
persist_notify_job_updated_already_persisted()) and call it from
resync_resolve_round() so HOOK_DETAIL_WORKER_UPDATE fires without a
spurious WAL append. TDD: new spy-hook regression test, RED confirmed
against the unfixed code, GREEN after the fix. Full controller_unit_test
suite: 1220 Tests, 0 Failures, 0 Ignored.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live 3-node HA soak crashed xce_controller: SIGSEGV in
peer_ctrl_msg_handler_inline (ha.c:7035) dereferencing a NULL
ep->data. gdb analysis of the actual core dump showed the endpoint
itself fully alive (refcnt=191964, tearing_down=0) while ep->data had
been synchronously freed and nulled by standby_ep_err_handler_primary,
the endpoint's err_handler, with zero coordination with the vtask pool
that was concurrently dispatching a peer-control message for the same
endpoint (g_drive_async_peerctrl_dispatch_enabled defaults ON).
The sibling handlers (vote_ep_err_handler, pre_vote_ep_err_handler)
already avoid this exact hazard, with an explicit comment: "Do NOT
free vctx here -- peer_ctrl_msg_handler may still reference ep->data."
standby_ep_err_handler_primary was never updated to the same
discipline when the vtask-pool dispatch machinery (bug-1197/bug-1298)
was introduced.
Fix: route standby_ep_err_handler_primary through the same
already-tested vote_ep_teardown_async() deferred, per-endpoint-FIFO-
serialized teardown that vote-endpoint demotion already uses, plus an
idempotency guard (endpoint_is_tearing_down) since the old "ep->data
== NULL" reentry check no longer applies once the free is deferred.
Also fixes a second-order double-unref this surfaced in
on_standby_connected's stale-duplicate eviction path, caught by ASAN.
TDD: two new tests prove the teardown is deferred until an in-flight
same-ep vtask completes, and that a repeated error callback on the
same endpoint is a safe no-op. Full controller suite: 1219/1219, zero
regressions.
Also logs bug-1840 (a separate scheduler heap-corruption crash from
the same soak run) as not-fixed pending an ASAN reproduction -- a
non-instrumented core dump cannot localize glibc's malloc_consolidate
abort back to the actual out-of-bounds write.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sim_reconcile_unaccounted_jobs() (sim_loop.c) was calling
picker->picker_iface->on_dispatch() directly from the sim driver thread,
violating plugin.h's documented "allocator thread only" contract for
Post-allocation hooks. Dormant in practice only because builtin_fcfs.c
happens to lock internally around every hook -- a defensive measure the
contract text doesn't actually require of picker plugins.
Since scheduler-sim-live-allocation, sim mode spawns the same real
sched_job_picker()/sched_allocator() thread pair normal mode uses, so a
real allocator thread now exists to route through. Added a
reconcile_req/reconcile_ack cbuf pair on xce_sched_core_t and a reserved
XCE_SCHED_RECONCILE_WAKE sentinel pushed through jbuf to wake the
allocator thread without touching jbuf's pinned capacity-1/item-type
invariant (Req 5 AC 7). sched_allocator() services the request and calls
on_dispatch() itself; sim_reconcile_invoke_on_dispatch() blocks on the ack.
Falls back to a direct call when no allocator thread is live for a given
core (standalone unit tests), which is safe by construction.
TDD: new test spawns a stand-in allocator-relay thread and asserts
on_dispatch() fires there, not on the caller's thread -- confirmed RED
against the old direct-call code, GREEN after. Four other test files with
hand-rolled sched_allocator() stubs needed the same relay branch (found by
an actual hang at meson's timeout). Full scheduler suite: 85/85, zero
regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:
- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
Reconciliation" backstop) force-applied any recorded DISPATCH decision
it found, with zero live capacity check. Added
sim_reconciled_alloc_fits_capacity(), reusing the same
sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
the live picker/allocator path already uses.
- bug-1838: even with that gate, reconciliation still preempted every
freshly-PEND job before the live picker/allocator got a turn each round
-- unnecessary now that scheduler-sim-live-allocation gave sim mode the
same real, capacity-checked picker/allocator threads normal mode uses.
Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
2026-09-11 widening whose own justification no longer holds), so a
freshly-pushed job now always flows through the same live, verified
decision path as normal mode. Reconciliation now only ever fires for a
job already RUN from before the replay's own observation window --
the one case structurally invisible to the live picker.
Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.
Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.
Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CtrlHelloRequest.key was declared `Data` in controller.capnp. This
project's capnp-c backend silently no-ops encoding of Data-typed fields
without zero-initializing the caller's local struct, leaving an
uninitialized capn_ptr that write_CtrlHelloRequest unconditionally wrote
to the wire -- SIGSEGV in capn_setp/copy_ptr. Hit live: ctrl-1 won its
Raft election and crashed calling ha_send_ctrlhello_to_all_standbys().
Same bug class as bug-1504/bug-1505, both already fixed by switching the
field from Data to List(UInt8); bug-1505's own writeup had flagged
CtrlHelloRequest.key as a known-but-unfixed instance. Applied the
identical fix here: Data -> List(UInt8) + keyLen companion in the schema,
uint8_t*/key_len in typemap.h. The field is unused in production (always
NULL), so no consumer code changed. Added a round-trip regression test
covering both the NULL-key production shape and real key content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
update_limit_res_value() computed a percent-based limit's capacity from
the resource's own live, fluctuating value (e.g. "mem"), letting the
enforced cap drift with unrelated memory pressure instead of staying
pinned to a stable fraction of total capacity. instantiate_limit() now
resolves a static capacity-source resource once at instantiation time
(mem->maxmem, swap->maxswap, table-driven), stored on a new
xce_vres_t.percent_res field; a missing counterpart falls back to
today's behavior rather than failing instantiation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds an if/then JSON Schema (draft 2020-12) conditional constraint to
limits.schema.json's Limit definition: when percent is true, value
must be a number in [0.0, 1.0]. This is the class of authoring mistake
bug-1831/bug-1832 fixed manually (night_mem_pct: value: 50 instead of
0.5) -- now caught automatically at real `xceadmin config build` time
(the actual cluster.bin generation path deploy-ha.sh uses), not merely
by a unit test.
Empirically verified against the real jsonschema library this repo
uses: a violation surfaces with the exact same file/json_pointer/
constraint contract validate_yaml()'s existing ValidationError already
provides, so zero changes were needed to its error-reporting code.
While implementing this, the new constraint caught an identical
pre-existing defect in xceadmin's own checked-in "full" golden test
fixture (fixtures/full/configs/limits.yaml, night_mem_pct: value: 50)
-- fixed the same way, and regenerated the checked-in expected.bin
golden artifact to match the corrected input.
Added 3 tests to test_validation.py: reject an out-of-range percent
value, accept an in-range one, and confirm a non-percent value is
never range-checked regardless of magnitude.
Build gotcha discovered and documented (cerebrum.md): this repo has
two copies of every bundled schema -- src/configs/schema/ (canonical)
and src/xceadmin/src/xcecommon/schema_files/json/ (a gitignored local
mirror meson-build.sh always overwrites from the canonical copy on
every build). Only the canonical copy needed a source-controlled edit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
fmt_limit()/fmt_limit_units() formatted a percent-based limit's value
as a bare snprintf(..., "%f%%", limit->val.val) -- echoing the raw
stored value with a literal "%" appended. But xce_limit_val_t.val is
genuinely a 0-1 fraction when percent is set (confirmed end-to-end:
no /100 or *100 scaling exists anywhere in the config->capnp->runtime
pipeline; confirmed authoritative by the original hand-written C
literal this codebase's oldest percent-based limit was migrated from,
.val = 0.2 for a documented "20% mem cap"). A correctly-configured 20%
limit therefore displayed as "0.200000%" instead of "20.000000%" --
misleading, though technically an honest echo of the stored value.
Fixed by multiplying by 100.0 before formatting. Display-only --
update_limit_res_value()'s actual capacity computation was already
correct and is untouched. fmt_limit() (a thin XCE_UNITS_RAW wrapper)
inherits this fix for its own percent-branch output too, a deliberate,
intentional change from its pre-fix behavior for percent-based limits
specifically (its 5 pre-existing direct tests only assert rc > 0,
never the exact percent string, so this is safe).
Updated test_limit.c's percent-mode test fixture from an unrealistic
.val=80.0 to a realistic .val=0.8 (assertion text unchanged, since
0.8*100=80 either way) and test_repl_limit_table.c's own percent test
to expect its shared fixture's arbitrary .val=50.0 constant correctly
rendering as "5000.000000%". RED confirmed via the raw Unity testlog's
:FAIL: lines (the aggregate meson summary masks single-test failures,
a known harness quirk in this project); GREEN confirmed after, both
lib and scheduler suites, 0 Unity FAILs.
Discovered during live manual verification of scheduler-sim-diagnostics
Requirement 6 against a real deployed cluster.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
User's own independent retest against a real deployed cluster produced
a fully populated, correctly human-formatted mem-based limit usage
line (limits2: accounted=18022400.000000 (17.19 MB)
capacity=4635344384.000000 (4.32 GB)) -- directly contradicting the
original entry's "never populates" framing, which was based on a
single short-lived agent-run live test.
Downgraded from a confirmed defect to an unreproduced, timing/
session-shape-dependent observation. The original single-run finding
was real and remains unexplained, but a categorical "mem-based limits
don't populate usage via replay" claim is now known to be false.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered during live manual verification of scheduler-sim-diagnostics
Requirement 6 (human-readable info limit formatting) against a real
deployed cluster with a real captured replay. A mem-based limit
(limits2/ug1_global_mem/night_mem_pct) never shows a live usage
instance even when its matching job is genuinely dispatched with an
intact rusage[mem=...] reservation -- while a slots-based limit
(limit1) in the same replay correctly populates one. Suspected timing/
ordering gap between WORKERS_PUSH replay-event application and
match_job_global_limit_()/match_job_host_limit_()'s resmgr lookup, not
confirmed. Out of Requirement 6's boundary (display-formatting only,
proven correct independently) and out of bug-1826's already-closed
scope (matching invocation itself, not this resource-type-specific
timing gap). Needs its own dedicated investigation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
core_clock.c joined the mechanically-enforced isolation archive
(sched_core_isolation_lib) and its own doc comment claimed coverage by
the forbidden-headers/forbidden-symbols text scans, but was actually
absent from both -- only the nm(1) undefined-symbol layer covered it,
which cannot catch a forbidden #include resolving to no undefined
symbol (a macro-only header).
Added XCE_CORE_CLOCK_C_PATH (meson.build, mirroring the existing
per-file path macros) and extended both
test_core_never_includes_forbidden_headers() and
test_core_never_calls_forbidden_symbols() to scan core_clock.c. It
remains excluded only from the wall-clock scan, for its one sanctioned
clock_gettime() call.
TDD: confirmed RED by temporarily inserting a forbidden #include into
core_clock.c (84/85, correctly attributed failure), reverted
(byte-identical to HEAD), reconfirmed GREEN (85/85). No live violation
existed -- this closes a coverage gap in the mechanism itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Same pattern as bug-1492/1682/1611: tags claimed "fixed"/
"closed-by-scheduler-sim-live-allocation" while the fix narrative said
"NOT FIXED" -- and correctly so, that feature never touched
test_core_isolation.c. Removed the incorrect tags; entry remains open.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The dimension-1 audit (previous commit) reported bug-1611 as only
"partially fixed" -- but that assessment worked from bug-1611's own
stale narrative text without checking scheduler-sim-eager-reconciliation
(a separate spec, completed 2026-09-11, predating this session's audit).
That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger
from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in
the same round it is recorded, never deferred to a later confirmation
event -- closing bug-1611's original motivating scenario for good.
The "residual gap" my own first correction pass invented (an
already-RUN-at-snapshot-load job) is also not real: such a job's
allocation comes from the snapshot directly (confirmed field-complete
by this session's dimension-4 audit), so it's never a reconciliation
candidate in the first place.
Corrected bug-1611's buglog entry to FIXED and updated
scheduler-driver-equivalence.md's stale "Known violations" section
(still described both bug-1611 and bug-1613 as open) to reflect current
reality, with a pointer to re-verify rather than re-trust either framing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
/xce-audit-invariants scheduler (dimension 1, driver state-equivalence,
re-run after scheduler-sim-live-allocation) found plugin_end_session()
still checked `plugin->begin_session_fn != NULL` before invoking
`plugin->end_session_fn(plugin)` -- a copy-paste of
plugin_begin_session()'s own (correct) guard. Two failure modes: a
plugin registering only end_session_fn silently never has it invoked;
a plugin registering only begin_session_fn crashes via a NULL
function-pointer call. Currently inert in production (no live plugin
hits either shape) but a real latent defect in exactly the
session-boundary code the driver-equivalence invariant governs.
This exact defect had been independently filed twice: bug-1492
(2026-08-04, scheduler-core task 5.2's review) and bug-1682
(2026-08-24, found adjacent to bug-1681's fix) -- neither investigation
cross-referenced the other. Both entries also carried a "fixed" tag
inconsistent with their own "not yet fixed" narrative text, caught by
this audit's independent code+text cross-check.
Fixed the one-line guard; added test_end_session_invokes_end_session_fn_
when_only_it_is_set() (test_plugin_hook_dispatch.c) proving the silent
non-invocation, confirmed RED (84/85) against the buggy guard and GREEN
(85/85) after the fix. Updated both buglog entries to FIXED with
cross-references, and corrected bug-1611's stale narrative to reflect
its narrowing (not elimination) by scheduler-sim-live-allocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds the feature's concrete original acceptance test: a genuine
capnp-encoded .replay (real ReplayHeader/SnapshotEntry/Conf4Sched with
one global and one host-scoped limit), driven end-to-end through the
real, unmodified sched_sim_driver_run(). The dispatched job's usage
registers against both limits via limit_get_configured_limits()/
limit_get_usage_instances() -- the same accessors info limit reads --
proving real, non-zero usage where today's REPL previously showed none.
RED-phase evidence: reverting task 4.3's 4 files to their pre-fix
commit reproduces the exact predicted failure (host-scoped usage
missing, global usage intact); restoring to HEAD reproduces GREEN.
Closes bug-1826 (the bug this whole feature was built to fix) --
buglog entry updated from NOT FIXED to FULLY FIXED, citing this test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Fully closes bug-1829's own tracked follow-up from task 4.1: retypes
seqjob_allocator.c/parjob_allocator.c's init_fn AND runtime hooks
(prepare_allocation_hook, job_alloc_fn) to read the plugin's own
honest plugin->core field (task 1.2) instead of deriving an
xce_scheduler_t* from plugin->sched -- mirroring task 2.3's limit.c
migration exactly. The gap was broader than initially scoped: not
just init_fn, but every runtime hook independently derived sched from
plugin->sched on each allocation attempt.
Adds seqjob_allocator_new_plugin()/parjob_allocator_new_plugin()
factories (plugin.h, mirroring limit_new_plugin()), letting
sim_alloc.c's bootstrap functions register both plugins DIRECTLY
against &(sim->core) via the existing plugin_register_core() -- the
same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already
use. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the
plugin-registry/fan-out path entirely; plugin_prepare_allocation()'s
fan-out during a live seqjob/parjob dispatch now genuinely reaches
sim->core.plugins, the same registry the limit plugin is registered
in.
One narrow, necessary exception remains, independently verified
confined: my_alloc() still needs a small xce_scheduler_t wrapper
(only .core.resmgr aliased to sim->core) to satisfy sched_make_allocs()
(scheduler.c, out of this task's boundary, unmodified) -- set on
plugin->sched only AFTER plugin_register_core() has run, never read by
the plugin registry or any prepare_allocation_hook fan-out.
TDD: new test_sim_alloc_limit_fanout.c drives a real live dispatch
(sched_job_picker()/sched_allocator() thread pair) with a host-scoped
limit registered directly against sim->core, proving a real usage
instance is created -- isolated from both reconciliation and
global-limit matching. RED: dispatch succeeded but the usage-instance
iterator was empty. GREEN: 82/82, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Replaces sim_picker_thread_fn() (the always-PEND placeholder) with the
real sched_job_picker()/sched_allocator() threads (task 3.1), spawned
against &(sim->core) in sched_sim_driver_run(). Simulation mode now
genuinely uses the jbuf capacity-1 rendezvous for the first time
(confirmed still exactly one producer/one consumer). Adds
sched_election.c to sched_sim_isolation_lib.
Fixes bug-1829 (blocking, only reachable once sim_allocator() actually
called sched_make_allocation() for the first time): core->seqjob_
allocator/parjob_allocator were never bootstrapped for simulation mode
at all. Fixed via new sim_alloc.c (sim_bootstrap_seqjob_allocator()/
sim_bootstrap_parjob_allocator()), mirroring the sim_picker.c/
sim_limit.c precedent.
That fix's own workaround -- a throwaway xce_scheduler_t wrapper
satisfying seqjob/parjob's init_fn -- has a documented, narrow side
effect: prepare_allocation_hook fan-out during a live seqjob/parjob
dispatch attempt reaches the wrapper's own separate plugin registry,
not sim->core.plugins, so a plugin registered only against the real
sim core (the limit plugin) isn't reached for host-scoped matching
during a live dispatch yet. Documented in design.md's new "Known Seam
Gap" section; task 4.3 added to tasks.md to close it before task 5.2's
acceptance test can pass for the host-scoped case.
TDD: new end-to-end test drives a real .replay with an empty decisions
array (ruling out reconciliation) through the real thread pair,
proving a genuine DISPATCH from live resource availability. 25
pre-existing white-box sim/repl test files given link-satisfying
stubs reproducing the OLD always-PEND contract unchanged -- their own
assertions are untouched (task 5.3's job to reconcile deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Migrates match_job_global_limit_()/limit_job_election()/
match_job_host_limit_() (plugins/limit.c) to read plugin->core (added
task 1.2) instead of deriving xce_scheduler_t* from plugin->sched,
which stays NULL for a plugin registered via plugin_register_core()
(simulation mode). Removes the dead ctx.sched write flagged during
task 2.1's review -- verified this ctx never has .sched read anywhere
downstream.
Fixes bug-1828 (found while writing this task's own TDD test for a
previously untested code path): match_job_host_limit_() had the same
key_find()-negative-rc-treated-as-fatal defect bug-1786 already fixed
in its global-limit sibling, but never applied here -- meaning
host-scoped resource limits were never actually enforced, in any
mode, until this fix (not merely a display gap; the limit's vres was
never added to obj->hlimits, so sched_fill_limit_requests() always
saw an empty list for it).
TDD: new tests drive the real hook-registered entry points against a
plugin_register_core()-registered plugin, asserting real usage
instances via limit_get_usage_instances() for both global and
host-scoped limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
info limit was correctly implemented and tested but inert against a real
replay: register_limit()'s only call site was normal-mode sched_register_
plugins(); sim mode's own plugin bootstrap (sim_bootstrap_picker()) never
registered the limit plugin. The snapshot format already carries the
original session's limit config (sim->core.config, a borrowed reference set
by xce_sched_core_load_snapshot()) -- nothing needed to change in the
recording/replay format.
New sim_bootstrap_limit_plugin() (sim_limit.c, mirroring sim_picker.c's
structure) registers the plugin via plugin_register_core() and seeds it via
a new limit_seed_config(plugin, xce_sched_core_t* core, config) entry point,
wired into sim_loop.c alongside the existing sim_bootstrap_picker() call.
limit_new_plugin() extracted from register_limit() as a factory, mirroring
builtin_fcfs_new_plugin()'s established pattern; register_limit() itself is
behavior-preserving.
First attempt set plugin->sched to a disguised xce_sched_sim_driver_t*,
relying on it sharing xce_scheduler_t's first-member layout to reuse the
existing config_hook path -- rejected on review: this repeats the exact
cross-cast pattern that already corrupted memory once in this codebase (see
core.h's xce_core_cache_clear_cb_t history). Fixed by making
limit_seed_config() take xce_sched_core_t* as an honest parameter instead,
with limit_update_config() (normal mode) and the new sim path both
delegating to a shared limit_seed_config_impl() -- no cast, no aliasing risk,
independently verified by two review rounds reading the actual struct
layouts and call chains.
Also added sim_limit.c to sched_sim_isolation_lib so Req 5 AC 17's
controller-I/O-isolation check covers it (an initial gap, closed in the same
remediation round). Two ripple waves of the same established link-stub
pattern applied to the ~24 white-box test files that #include sim_loop.c.
info limit now shows real configured-limit data against a genuine replay.
Live usage will still correctly show "no active usage instance yet" until a
separate, larger fix (logged as bug-1826) wires limit *matching* into the
sim decision path -- deliberately out of this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified while closing out scheduler-sim-
diagnostics task 5: register_limit() has exactly one call site in the whole
codebase (normal-mode sched_register_plugins()); sim mode's own plugin
bootstrap (sim_bootstrap_picker()) only registers the picker plugin. The
info limit command built across this spec's 5 tasks is correctly
implemented and thoroughly tested but currently inert for a real operator
running xce_scheduler -i/-s <replay> -- it will always report "no limits
configured" regardless of the original recording's actual configuration.
Not fixed -- out of task 5's test-only boundary; recommended as the next
piece of work for this open-ended spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified (implementer + reviewer, both reading
src/lib/zarray.c directly) while implementing scheduler-sim-diagnostics task
3.1. Not fixed -- out of that task's scope; its own new code avoids the
mistake rather than touching the pre-existing function.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
