gitea-sync.py: --ids syncs named bugs regardless of dates (OR'd with
--since). Bulk triage bumped last_seen/status_updated across most of the
buglog, so --since 2026-09-01 matched ~1150 bugs while recently fixed older
bugs (bug-1798..1804, 1714, 1754, 1850, 1854) could never be reached.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.
- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
stages only; a POST_EXEC code is the script's and leaves exit_info alone.
Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
-errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
real transition, as apply_committed_entry() already does.
New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every scheduler (re)join re-pushes the whole config, and both apply paths
-- sched_init_resources() (scheduler.c) and core_apply_config_resources()
(core_events.c) -- re-added each resource with makeResource(), which
refuses an existing name: one ERROR per resource per rejoin (33 in
data/xce_scheduler.log), and a genuinely changed definition was dropped
just as silently.
New libxce resourceApplyDef(): under one write lock, register a new
resource (ADDED), or compare an already-registered one on its
scheduling-relevant fields -- valtype, ordtype, flags, interval -- and
return SAME or CHANGED. Both call sites log SAME at debug, CHANGED as a
WARN that the definition is not applied to the running scheduler
(updating a registered resource in place would disturb its index and
accounting), and only real failures as ERROR.
Test test_resource_reapply_same_is_noop_changed_is_reported (EEXIST on
an identical re-push before). Scheduler suite 87/87, lib 1/1. Re-verify
of data/scheduler-replay-20260925-133708.replay: 0 add-resource ERRORs
(33 before), 0 changed-definition WARNs, 0 mismatches, same 60 DISPATCH /
908 PEND decisions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_failover_classify() mapped E_XCE_NOT_READY with NOMASTER to
RETRY_BACKOFF, so xcecli_retry_next() rotated to the next master,
reconnected and fired the rejoin hook. But a NOT_READY reply comes from a
controller that answered on a working connection -- here a healthy primary
that had briefly lost its write quorum (both standbys NACKing, bug-1864).
The scheduler abandoned it for the standbys, rejoined, and got a full
re-push built from lagging committed state (21:39:24-27 in data/logs).
New action XCE_FO_RETRY_SAME (appended; existing values unchanged):
NOT_READY backs off (same jittered schedule, same -E_XCE_NOT_READY on
budget exhaustion) and re-sends on the current connection -- no rotate,
reconnect or rejoin. NOMASTER keeps rotating after backoff. If the
controller is really a standby it answers REDIRECT/STANDBY/NOMASTER next
time, and if it dies the transport error rotates, as before.
Tests (test_libxce_retry.c): taxonomy row NOT_READY -> RETRY_SAME;
decide backs off and exhausts to -E_XCE_NOT_READY; the real
xcecli_retry_next keeps cur_master on NOT_READY (rotated before) while
STANDBY still rotates; backoff-growth test moved to NOMASTER. Failover
tests 56/56; lib, controller 5/5, worker 10/10, scheduler 87/87,
harness 5/5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_ctrlhello stored the netio endpoint's fd (primary_ctrl_fd = ep->fd)
and ha_on_primary_ctrl_fd_rdhup close()d it in three branches. netio owns
that socket and closes it on endpoint teardown, so by the time the
liveness poll fired the RDHUP the number could be reused -- in a chaotic
soak by the standby's WAL segment: HA's close() shut the WAL, fdatasync
failed 2305 times (EBADF, then EINVAL once a socket took the number),
durable_lsn stuck, and one round's jobsub failed EIO.
handle_ctrlhello now also holds a reference on the endpoint
(primary_ctrl_ep), which keeps its socket open and its fd number ours.
ha_drop_primary_ctrl_conn() replaces the raw closes: shutdown() the
still-owned socket so netio sees the hang-up and tears it down itself,
then release the reference. A bare fd with no endpoint is only forgotten.
Test test_rdhup_never_closes_an_fd_netio_already_released: real
ctrlhello -> netio teardown -> file opened on the freed number -> RDHUP;
the file was closed before, survives now. test_rdhup_closes_and_nulls_fd
updated to the new contract (peer reads EOF, fields cleared). Controller
suite 5/5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The dispatcher checked each queued task's endpoint busy flag one item at a
time, while the vcpu that finished the endpoint's previous message cleared
the flag without the taskq lock (a lock there deadlocked, bug-1282). Within
one scan, task N was skipped (endpoint still busy), the endpoint freed up,
and task N+1 of the SAME endpoint was dequeued first. Standby controllers
then handled WAL catch-up entry 258 before 257 and NACKed (log mismatch):
26 reorders -> ~576 NACKs in a 45-min production-like run, both standbys
at once at 21:39:24, primary lost its write quorum and answered
jobdispatch/join with E_XCE_NOT_READY, the scheduler client failed over and
rejoined, and the rejoin JOBS_PUSH carried lagging committed state
("refusing resync state ... existing job is RUN" WARNs).
dataq gains dataq_deq_ctx() (a per-scan matcher context). The scan is now
vtask_deq_eligible(taskq, is_blocked): it remembers every endpoint found
busy and keeps all later tasks of that endpoint ineligible for the rest of
the scan (past 64 distinct busy endpoints, every later IO task waits for
the next scan) -- per-endpoint FIFO without any new lock.
Test lib/testsuite/test_vtask_order.c injects is_blocked to free the
endpoint mid-scan: task N+1 was dequeued before the fix, none after; other
endpoints and FIFO/generic tasks unaffected. Suites: lib 1/1, controller
5/5, worker 10/10, scheduler 87/87, harness 5/5. Chaotic HA soak 2/2 PASS
with 0 WAL reorders (2 and 5 per run before), no ASAN.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a later primary of a COMPOUND failed, sched_make_compound_alloc_
still moved the earlier primaries' merged allocation onto the schedobj,
so a job decided PEND carried a partial allocation that the next
successful round overwrote and leaked. The merged allocation is now
published only when every primary succeeds and freed otherwise. No
resource-state rollback is needed: the primaries' uncommitted debits live
in ctx.uncomitted, per-attempt scratch freed by sched_ctx_fini(), and
schedobj->alloc's only reader is the dispatch path. The ALTER loop's
per-alternative cleanup (added with bug-1769) is now redundant and removed.
Test: "1*{..} + 5*{..}" on 2x2 slots PENDs with schedobj->alloc NULL
(held the first primary's allocation before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The 16 commits since 534895c5 were re-signed (rebase --exec amend -S; tree
unchanged), which changed their hashes. Update the buglog and the
scheduler-allocation completion note to the new ones.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each primary's allocator (seqjob/parjob my_alloc) sets ctx->alloc to a
fresh job allocation covering only its own hosts, so a multi-primary
COMPOUND ("1*{A} + 2*{B}") dispatched only the last primary's hosts --
leaking the earlier allocations while every primary's hosts had been
debited in ctx.uncomitted.
sched_make_compound_alloc_ now folds each primary's allocation into one
job allocation via the new libxce merge_job_alloc(): a host placed by
several primaries becomes one entry with per-resource values added
(matched on resname, instid and reserve, so distinct instances are never
summed); new hosts and shared allocations are appended/merged the same
way. Partial-failure rollback remains bug-1766.
Tests: "1*{..} + 2*{..}" on 2x2 slots now allocates 3 slots across two
distinct hosts; "1*{..} + 1*{..}" yields one host entry with 2 slots
(both kept only the last primary before). Scheduler suite 87/87, lib 1/1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seqjob_prepare_allocation() and parjob_prepare_allocation() returned 0
whenever ctx->schedobj was not PRIMARY, so a primary nested in a COMPOUND
(or ALTER) job never had its slot count or rusage checked: a 5-task
compound dispatched onto 4 slots, and an ALTER job was "placed" by its
first alternative with an empty (0-slot) allocation. Both hooks now
compute the want from ctx->primary -- the primary being allocated, set by
sched_ctx_set_primary() for nested primaries and identical to
obj->u.primary->resreq for PRIMARY jobs.
sched_ctx_set_primary() also clears ctx->worker, so the next primary of a
compound computes its job-level want rather than a per-host want left
over from the previous primary's host (PRIMARY jobs set it once, with
worker already NULL).
Tests (real ingest + live allocator): a 5-task COMPOUND on 2x2 slots now
PENDs; "{5*{..}} || {1*{..}}" falls back to the second alternative (1
slot). Both failed before. Scheduler suite 87/87.
Follow-up logged: bug-1863 (multi-primary COMPOUND keeps only the last
primary's allocation).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1769: every builder stores xce_compound_schedobj_t* (24 bytes) in
obj->u.alters, but sched_make_alter_allocation() read them as
xce_sched_obj_t* (176 bytes) and ran the compound allocation on them -- a
heap-buffer-overflow read of ->u.compound and write of ->alloc for any real
`{..} || {..}` resreq. The compound allocation now takes its resreq
explicitly (sched_make_compound_alloc_), and each ALTER alternative is
tried on behalf of the ALTER obj itself; a failed alternative's partial
allocation is dropped before the next (rollback itself is bug-1766).
sched_ctx_init() rejected ALTER objs ("bad sched obj type"), so ALTER jobs
could never allocate at all -- it now accepts them like COMPOUND (all
downstream checks only distinguish PRIMARY).
bug-1771: plugin_commit_decision() fans every DISPATCH out to the legacy
"xce_sched_fcfs" plugin even when it never tracked the job; fcfs_del_job()
(and fcfs's job-add hook) dereferenced job->queue->name with no NULL
check, SEGV'ing the allocator for a job whose queue was never registered.
Tests (test_parjob_e2e_multislot.c, real ingest + live allocator thread):
test_alter_resreq_job_dispatches_via_real_allocator (ASAN
heap-buffer-overflow before, DISPATCH after) and
test_dispatch_of_job_with_unregistered_queue_does_not_crash (SEGV at
fcfs.c:318 before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All 16 subtasks were already done; check the 6 parent tasks and set
phase implementation-complete. Verified 2026-09-24/25: scheduler suite
87/87 (ASAN), live multi-host e2e PASS, HA soak sequential + chaotic PASS.
Out-of-scope follow-ups stay open: bug-1766, bug-1768, bug-1769,
bug-1771, bug-1860.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The main loop adds and reaps worker-list entries (add_worker,
cleanup_worker_list -> remove_worker) while controller request handlers on
vtask threads walk the same list with no lock. A rejoin resync overlapping
a job reap read a freed node (ASAN heap-use-after-free in
find_worker_by_jobid), killing the worker mid chaotic HA soak.
- xcecd_worker_list_lock/unlock: add_worker links a fully built entry under
it; remove_worker unlinks under it and frees after.
- handle_jobstatequery holds it from reading the head through the query
page (the report send is a non-blocking xce_request); job control and
remote task start copy what they need under it instead of keeping the
entry pointer.
- The finished-job retention store (appended by the main loop, read by
resync) gets its own mutex around its public entry points.
Test: test_build_report_safe_against_concurrent_list_mutation -- UAF 3/3
with no-op lock stubs, clean 5/5 with the fix; worker suite 10/10; chaotic
HA soak 2/2 PASS, 6/6 resync jobs reconciled, no ASAN reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The finish notify is fire-and-forget, so the retained outcome is the only
way a controller learns a finish lost in a failover. Its 60s window ran
from the finish time, so an 84s outage expired it before the rejoin
resync could report it; the controller got "unrecognized" and requeued a
completed job (then stuck: scheduler kept RUN, see bug-1860).
Expiry is now measured from max(finish_time, last reconnect)
(xcecd_on_reconnect), and nothing expires while checking_connection's
hello probe reports the connection unhealthy. The reported finish_time_ms
is unchanged.
Verified: worker suite 10/10 (two new retention tests, red first); chaotic
HA soak 3/3 PASS (was 1 in 3 failing).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failing TEST_ASSERT longjmps out and skips the fixture teardown, leaving
the commit engine's sweep thread and persist's fsync_worker asleep on a
static struct. The next test re-inits it underneath them, corrupting the
condvar, and a later teardown blocks in pthread_cond_destroy/join until
the 300s meson timeout. The failure line itself was lost because stdout
was fully buffered.
- controller_test: line-buffer stdout; tearDown() reaps what a failed test
leaked via new XCE_TESTING registries (persist_test_reap_unstopped, then
commit_test_reap_unfinalized_engines -- persist_stop before engine fini).
Clean run reaps 0; an injected failure now fails in 2s instead of hanging.
- persist: flusher_flush_once reports only its own durable advance, so
fsync_worker no longer re-fires the durable callback for an advance a
concurrent persist_flush made (flaky "must not re-fire" test). Regression
test test_flusher_pass_does_not_claim_concurrent_flush_advance.
- tests: jap observer is atomic and waits (bounded) for a continuation that
may resolve on the flusher thread; pump catch-up tests wait for trailing
replies instead of a single scan.
- config_repl: upgrade decode temp file via mkstemp instead of a
pthread_self()-derived /tmp name (same value across processes, opened
without O_EXCL -- collisions and a symlink target).
Verified: controller_unit_test 1222/1222; 24 parallel runs (6x4), no
hangs or ASAN errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
q->resreq holds an interned makeString() value (pointer past the xstring
object header). core_upsert_queue() and sched_update_existing_queue()
released the old value with FREEUP()/free() when a queue update carried a
new resreq, an ASAN bad-free. Seen as HA soak replay-parity errors=1: the
sim replay of a snapshot-restored queue followed by a config push aborted
in core_upsert_queue.
Both sites now dereference() the old value (NULL-guarded). Regression
tests cover each site; the existing queue-update test had skipped the
resreq branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since bug-1790's fix (9caf1424) counts a job's full task total against
global limits, the repo's limit1 (all users except rshen share 3 slots)
pends xce_test's root-submitted 33-task job forever, so the multihost
e2e fixture failed even though the allocator is correct.
deploy-ha.sh gains an optional XCE_DEPLOY_CONFIG_DIR (default unchanged:
src/configs). run-multihost-verify.sh copies src/configs into its deploy
dir, replaces limits.yaml with an empty list, and deploys with that
override.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- buglog: gitea_issue / gitea_synced_status for the 20 bugs created and the
30 already-synced entries (#25/#35 reopened as partial)
- .kiro/specs/*/spec.json: "gitea" milestone/issue mapping for all 66 specs
(65 milestones + 195 issues created, 158 closed at creation)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gitea-sync.py fixes found on the first live run (bug-1855 follow-up):
- 204/empty-body replies (label DELETE) no longer crash in json.loads
- every request has a 60s timeout, so a stalled call fails instead of hanging
- spec.json sync state is inserted as a trailing "gitea" block instead of
re-serializing the file (15 hand-formatted spec.json files would reflow)
- implementation issue body labels generated vs approved correctly; the
validation body uses the same done-phase set as the close rules
Project memory now routes all buglog writes through the buglog skill:
CLAUDE.md key rule, a cerebrum Do-Not-Repeat entry (with the older
duplicate-id / json.dump / tags-vs-fix lessons tagged as superseded), and
xce-audit-invariants step 5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog had no status field, so "is this bug open?" meant reading free
text, and gitea-sync closed any issue whose `fix` text was non-empty --
including "NOT FIXED"/"PENDING"/"PARTIALLY FIXED" ones (bug-1855).
- Backfill `status` (+ status_updated) on all 1,193 entries. The 85 entries
whose text said not-fixed/pending/out-of-scope were checked against HEAD
and git history; the rest are `fixed`. Result: 36 open, 10 partial,
2 investigating, 5 cannot-reproduce, 4 wontfix, 1 duplicate.
Insert-only diff: existing lines are untouched.
- .claude/rules/buglog-status.md defines the status values (kept out of
openwolf.md, which `openwolf update` overwrites).
- scripts/buglog.py + buglog skill: add (next free id), status, note,
recur, list, show, triage. Edits only the touched entry's lines; a full
json.dumps rewrite churns ~1,200 lines of mixed \u-escaping.
- gitea-sync (script + skill): state and a single bug:<status> label come
from `status`; reopen supported; issues shared by several entries follow
the most-open one; only status labels are swapped; per-entry write-back
with gitea_synced_status so unchanged entries cost no API calls; list-typed
`fix` no longer crashes; spec issues close on done phases (spec.json has
no implementation approval); --since no longer floods Gitea after the
backfill.
- Disable OpenWolf buglog auto-detection: its hook numbers new entries
len(bugs)+1, which collides with existing ids (bug-1856, upstream).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- .wolf/anatomy*, memory.md: hook-maintained index and session log
through the bug-1851..1854 work.
- .kiro/specs/libxce-alloc-preason-equality/: requirements, design,
research and tasks generated; none approved yet (spec.json phase
tasks-generated).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli_retry_next's "xcecli failover: attempt N -> host:port" line now
carries "(after status=<rc>)". Without it a log cannot tell a dead
connection from a healthy primary's commit-timeout or admission-backpressure
reply -- the distinction bug-1850 (the live scheduler re-joining every
~100s) needs to find its triggering rc. Diagnostic only; no behavior change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A chaotic round's walcheck compared all three members' on-disk WALs with
boundary=auto. The just-killed term-4 primary still held its NACKed,
never-committed lsn=58, which differed from the new term-5 leader's
own-term noop at lsn=58, and was counted as class=committed.
walcheck_count now takes an optional "midloop" mode, used only by the
per-round chaotic check: the boundary is the serving primary's own
committedLsn (live_committed_boundary() needs 3-of-3 convergence, which a
chaotic round never reaches), and only members guaranteed to hold the
committed prefix are compared -- the primary plus live standbys whose
appliedLsn has reached the boundary. Every other caller, including the
settled final sweep, still compares all three. Log lines name members=.
Also logs bug-1854 (scheduler re-adds every configured resource on each
rejoin config push; ERROR noise, changed definitions ignored) for
follow-up.
10-round chaotic soak: PASS (divergence 0, job loss 0, SIMVERIFY 9 files,
0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When the live worker + scheduler pair is up, the soak now starts the
scheduler with --record-dir, records (xceadmin scheduler record start)
from each scheduler (re)join until just before the next primary kill,
stops it cleanly so the .replay is flushed, and at the end replays every
clean recording through `xce_scheduler --sim --sim-verify`. Any decision
MISMATCH, a failed sim run, or zero clean recordings FAILs the soak:
[xce-test-ha] SIMVERIFY files=N mismatches=M errors=E unclean=U
Recordings that could not be stopped cleanly are counted as unclean and
never verified (their unflushed tail would give false mismatches), except
a stop whose reply was lost mid-failover: when the same scheduler pid
answers "not recording" and logged no recording abort, the first stop
landed and the file is kept. Mismatches are counted from both the sim's
log file and stderr (the logger creates its file lazily).
XCE_HA_SCHED_REPLAY_VERIFY=0 disables the gate.
First runs caught bug-1851 and bug-1852.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The recorder encoded each controller event batch after
xce_core_push_events() had applied it, and applying moves payload fields
out of the events: a WORKERS_PUSH's resvals (core_apply_worker_resvals
MOVE_PTR), a JOBS_PUSH's allocs, a JOB_RUSAGE's rusages, and a
CONFIG_PUSH's shared resvals (sched_apply_pending_config recorded after
sched_init_config). Recordings silently lost exactly those fields -- a
worker's resval names were written with NULL values, so --sim-verify
replayed "no slots" and turned 3 live DISPATCHes into PENDs.
driver.c now records the batch before the push; the periodic in-stream
SnapshotEntry moves out of sched_record_write_events() into new
sched_record_checkpoint_after_apply(), called after the push so a
checkpoint still reflects the events written ahead of it. The config
push is likewise recorded before sched_init_config().
Regression test: test_recorded_workers_push_keeps_resval_values (RED:
recorded value NULL; GREEN after). Periodic-checkpoint tests updated to
the write-then-checkpoint contract. meson test 109/109; 25 live soak
recordings replay with 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_val_res() memcpy'd the xce_val_t into w->host.resvals, so the
"type"/"model" string resources aliased the interned xstrings owned by
w->host.attrs. rejoin_worker() drops the old attrs strings, then
apply_join_state() -> resync_arm_round() resolves the previous join's
still-outstanding round, firing the worker hooks before
add_hostattrs_resvals() re-seeds resvals; the apiserver hook's
collect_resval() strdup()ed the freed hostmodel. Live: a newly promoted
primary died ~8s after election during an HA soak.
A string resval now stores a private strdup (taken before the old copy is
freed), and the owning arrays free their entries: host.resvals gets
kill_resval (replacing the FIXME NULL killer) and c->sharedinst's
resval_obj_killer (previously a no-op) frees the string and the value --
both also leaked every xce_val_t before. Also closes the same alias for
worker-reported string resvals and config values.
Regression test: test_rejoin_worker_string_resvals_survive_outstanding_round
(RED reproduced the live ASAN stack; GREEN after). meson test 109/109.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follow-ups from the bug-1846 investigation.
bug-1847: config pushes never reached a .replay recording -- handler.c
applied them synchronously, bypassing the pending-update queue the
recorder writes from. With bug-1846's deferral they're now applied at a
round boundary on the driver thread, so sched_apply_pending_config()
writes a CONFIG_PUSH event via the existing sched_record_write_events()
(no-op when not recording), ahead of the round's other recorded events.
Test: test_config_push_deferred.c's test_applied_config_push_is_recorded
(RED: 0 events; GREEN: 1). Fidelity limit, pre-existing: replay applies
configPush through core_events.c's CONFIG_PUSH path, which doesn't re-run
groupmgr_load_configs()/plugin_config_update(), so a limits/groups change
still won't take effect in sim.
bug-1848: xce_sched_obj_t.hlimits was created with a NULL zarray killer
at all three constructors, so reset_sched_obj() and schedobj teardown
leaked the per-worker xlist_t match_job_host_limit_() callocs. New
kill_hlimits_list() (schedobj.c) frees the list, never its vres entries
(owned by the limit plugin); core_events.c and snapshot_load.c use
identical static local copies, since both avoid sharing helpers with
schedobj.c and several test binaries link them without it. Test:
test_limit_host_dedup.c's test_hlimits_zarray_releases_its_lists (RED:
killer NULL; GREEN).
bug-1849: every --sim run warned "failed to resolve job picker plugin ...
picker rebuild will be skipped". Benign: the snapshot is loaded before
the sim registers its picker, and sim_bootstrap_picker() replays the same
queue/job walk afterward. Downgraded to TRACE with an explanation.
Buglog also records bug-1850 (open): the ~100s reconnect churn behind
bug-1846 part 2 -- xcecli_retry_next() rotates/reconnects on every retry
(ETIMEDOUT after the 10s reply deadline, E_XCE_NOT_READY, redirects), so
a slow reply from a healthy primary forces a rejoin + full re-push.
Needs the live scheduler.log and a libxce spec.
Full scheduler suite 87/87 (asan). Replay of the captured file: same 184
mismatches, all attributable to the recording's pre-fix binary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.
Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.
Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.
Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
-- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
returned NULL right after the push), GREEN (session walks the remaining
jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
#include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).
Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
match_job_host_limit_() (plugins/limit.c) runs on every per-host
allocation attempt via limit_prepare_allocation(), and unconditionally
xlist_store()d the matched vres into obj->hlimits[worker->idx]. That list
is only rebuilt by limit_job_election() on a configver bump, so a job that
kept pending on the same host gained one more copy per attempt.
sched_fill_limit_requests() then staged every copy into
ctx->alloc_req.host, and seqjob's make_host_alloc() built a host alloc
carrying all of them.
Found via temporary per-decision --sim-verify diagnostics (reverted, not
committed) against scheduler-replay-20260923-085356.replay: pc13v host
allocs held mem + slots + ~70 identical limit entries (72) instead of 3.
After the fix the replay side is always 3 entries, and the LIVE recording
itself still shows the inflation (72/81/96/114/138/177 entries, resetting
only on a config push) -- so this is a production bug, not sim-only.
make_job_account() creates one res_account per entry and
sched_update_res_accounts() charges each, so the host limit was charged
(and later released) N-fold.
Fix: hlimits_contains_() pointer-identity check (vres is interned per name
in privdata->instances); a matched vres is stored once per (job, worker).
Test: test_limit_host_dedup.c (sched_limit_host_dedup_test) -- five
plugin_prepare_allocation() attempts for one job on one worker must leave
hlimits[worker]->count == 1 and one staged copy of the vres in
alloc_req.host. RED before the fix (Expected 1 Was 2), GREEN after; full
scheduler suite 86/86 (asan).
Still open (bug-1846 part 2, logged in buglog): 9 live-recorded rounds end
after 11-12 decisions while replay processes the whole candidate list; not
root-caused, needs the live scheduler.log or a fresh capture.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/xce-audit-invariants scheduler dimension-1/4 audit, prompted by a fresh
user-captured .replay file that still showed massive --sim-verify mismatch
volume after bug-1843's fix. Root-caused via temporary diagnostic
instrumentation against the real capture (reverted, never committed) rather
than trusting the surface symptom.
bug-1844: sim_verify_alloc_eq() (sim_verify.c) compared `instid`, a bare
per-process monotonic counter (scheduler.c's sched_get_id(), core->nextid)
stamped at allocation-build time -- not semantic content. The original
recording's counter and a fresh replay's counter have no reason to reach
the same value for "the same" dispatch. Live-verified: every DISPATCH
mismatch showed identical value/duration/resname/reserve, differing only
in instid (e.g. 826 vs 1). Dropped from the comparison, same treatment as
bug-1842's `from` exclusion.
bug-1845: sim_verify_job_preason_eq() required PEND decisions' full
host/resource reason-detail lists to match. core.h documents "simulation
isolation" as deliberate design: sim/replay never registers
core->unsatisfiable_lookup_cb, so it can never hit the unsatisfiable-cache
main.c's normal-mode branch registers. A live cache hit
(sched_pend_from_cached_reason_()) records only main_reason with zero
detail; replay always falls through to the full per-host computation and
always produces detail. Same semantic outcome (PEND, same main_reason) at
two architecturally-legitimate detail levels. Relaxed the comparator to
main_reason only. Separately verified the cache itself is sound (key is
safe within a round since capacity only decreases and the cache clears
every round) -- ruling it out as the cause of a second, still-open
divergence (bug-1846: one job genuinely dispatches one round early in
replay vs. the original recording -- a suspected resource-accounting
parity gap, not yet root-caused, logged for follow-up).
bug-1772 (previously deferred, incorrectly tagged "fixed"): re-found
independently by a dimension-4 snapshot-field-completeness audit subagent.
snapshot_build_primary_schedobj() (snapshot_load.c) never computed
resreq_fp, unlike its declared model core_build_primary_schedobj()
(core_events.c) -- every snapshot-restored job got a zeroed fingerprint,
which alloc_cache.c's cache key folds in directly alongside
user/queue/project, so two distinct restored jobs sharing those three
fields could silently share an incorrect unsatisfiable-cache verdict.
Added the same 4-field computation the live path uses.
TDD throughout: each fix has a dedicated regression test, and each was
verified via genuine red (revert fix, confirm the new test fails) then
green (restore, full 85/85 scheduler suite). Also ran three parallel
dimension audits (core I/O isolation, picker/allocator jbuf rendezvous) --
both clean, no violations found.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sched_record_session_add_decision() (recording_session.c) grows its
per-round decisions array via realloc(), doubling from 8. On a failed
growth it silently dropped the current decision and every one after it
that round, without marking the round incomplete -- finalize() then
wrote the short array as an ordinary, apparently-complete ReplaySession.
--sim-verify's replay comparison had no way to distinguish that from a
genuinely smaller live round, producing spurious "expected: <none>"
mismatches for jobs that were, in reality, correctly decided live and
simply never recorded. Live scheduling itself was unaffected either
way -- only the .replay file's own record of that one round lost its
tail.
This refutes the leading theory from the prior investigation (replay
seeing a job's first appearance "one round early" via coarser event
batching in sim_loop.c) -- confirmed via targeted diagnostic
instrumentation against the original captured .replay file: replay
deterministically reproduces the full candidate set every round;
the recording itself has the one-round gap.
Fix: track truncation on the session ctx; finalize() now refuses to
write an incomplete session and fails loudly through the same
sched_record_abort_on_error() path every other recording failure in
this subsystem already uses.
TDD: added a regression test forcing a real, deterministic realloc()
failure via -Wl,--wrap=realloc (scoped to this one test binary --
no prior malloc/realloc fault-injection harness existed in this
codebase). Verified genuine red -> green: the test fails without the
fix, passes with it. Full 85/85 scheduler suite green.
Note: this prevents future recordings from losing decisions this way;
it cannot retroactively repair an already-captured .replay file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- .wolf/anatomy-index.json, anatomy.md, memory.md: OpenWolf hook-maintained
project index and session log, refreshed by this session's activity.
- .kiro/specs/controller-peers-env-fallback/: spec scaffold (brief,
requirements, design, research) from prior spec-init work.
- issues/new_requirements.md: scheduler simulator enhancement backlog
(info-command improvements, pend-reason detail, decision tracing).
Excluded as non-persistent: issues/Issues.md (empty placeholder),
src/lib/testsuite/test_xcectx_reconnect_race.c (self-documented as an
ASAN stress reproduction only, not a committed regression test, and not
wired into meson.build), scheduler-replay-20260922-095207.replay
(4MB binary debug capture, no .replay files are tracked in this repo).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sim_verify_alloc_eq() compared xce_alloc_t.from, which make_alloc()
(xcealloc.c) stamps with the real OS wall clock (time(NULL)), not
core.c's virtualized sim clock. The recorded (expected) value is
whatever real second the original run happened to execute in; the
freshly-rebuilt (actual) value is whatever real second the replay
executes in -- two unrelated real-world timestamps, so every
otherwise-correct DISPATCH decision failed verification whenever a
replay ran even one wall-clock second away from its own recording.
Drop `from` from the comparison -- it's reservation bookkeeping, not
part of a decision's semantic identity. Verified against a real
captured .replay file: 120/120 false DISPATCH mismatches -> 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Existing worker-rejoin-resync coverage (sequential/chaotic) only proves a
worker-dispatched job survives a real failover -- it never re-checks the
worker's own resource accounting afterward, which is exactly where
bug-1841 lived. This mode over-submits jobs to empirically discover the
worker's real concurrency ceiling (self-verifying via a PEND overflow
probe, not assumed from resval), fails over the primary while jobs are
RUN, waits for genuine rejoin + DONE, then hard-asserts the probe job
(still PEND from before the kill) now reaches RUN -- proving the
scheduler's resInUse actually unfroze.
Live-validated against a real 3-node cluster + xcecd_virtual worker:
settled at 3 RUN / 41 PEND (limits.yaml's limit1, not raw worker
capacity, was the actual ceiling -- exactly the kind of environment
detail this design avoids needing to assume), all 3 recovered to DONE via
genuine rejoin, probe reached RUN 219ms after recovery. Two earlier
design iterations are documented in the round's own comments: sequential
submit-then-wait let early jobs finish before late ones were even
submitted, and assuming resval's raw capacity as the concurrency ceiling
doesn't hold when queue/user limits bind first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
