gitea-sync.py fixes found on the first live run (bug-1855 follow-up):
- 204/empty-body replies (label DELETE) no longer crash in json.loads
- every request has a 60s timeout, so a stalled call fails instead of hanging
- spec.json sync state is inserted as a trailing "gitea" block instead of
re-serializing the file (15 hand-formatted spec.json files would reflow)
- implementation issue body labels generated vs approved correctly; the
validation body uses the same done-phase set as the close rules
Project memory now routes all buglog writes through the buglog skill:
CLAUDE.md key rule, a cerebrum Do-Not-Repeat entry (with the older
duplicate-id / json.dump / tags-vs-fix lessons tagged as superseded), and
xce-audit-invariants step 5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog had no status field, so "is this bug open?" meant reading free
text, and gitea-sync closed any issue whose `fix` text was non-empty --
including "NOT FIXED"/"PENDING"/"PARTIALLY FIXED" ones (bug-1855).
- Backfill `status` (+ status_updated) on all 1,193 entries. The 85 entries
whose text said not-fixed/pending/out-of-scope were checked against HEAD
and git history; the rest are `fixed`. Result: 36 open, 10 partial,
2 investigating, 5 cannot-reproduce, 4 wontfix, 1 duplicate.
Insert-only diff: existing lines are untouched.
- .claude/rules/buglog-status.md defines the status values (kept out of
openwolf.md, which `openwolf update` overwrites).
- scripts/buglog.py + buglog skill: add (next free id), status, note,
recur, list, show, triage. Edits only the touched entry's lines; a full
json.dumps rewrite churns ~1,200 lines of mixed \u-escaping.
- gitea-sync (script + skill): state and a single bug:<status> label come
from `status`; reopen supported; issues shared by several entries follow
the most-open one; only status labels are swapped; per-entry write-back
with gitea_synced_status so unchanged entries cost no API calls; list-typed
`fix` no longer crashes; spec issues close on done phases (spec.json has
no implementation approval); --since no longer floods Gitea after the
backfill.
- Disable OpenWolf buglog auto-detection: its hook numbers new entries
len(bugs)+1, which collides with existing ids (bug-1856, upstream).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- .wolf/anatomy*, memory.md: hook-maintained index and session log
through the bug-1851..1854 work.
- .kiro/specs/libxce-alloc-preason-equality/: requirements, design,
research and tasks generated; none approved yet (spec.json phase
tasks-generated).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A chaotic round's walcheck compared all three members' on-disk WALs with
boundary=auto. The just-killed term-4 primary still held its NACKed,
never-committed lsn=58, which differed from the new term-5 leader's
own-term noop at lsn=58, and was counted as class=committed.
walcheck_count now takes an optional "midloop" mode, used only by the
per-round chaotic check: the boundary is the serving primary's own
committedLsn (live_committed_boundary() needs 3-of-3 convergence, which a
chaotic round never reaches), and only members guaranteed to hold the
committed prefix are compared -- the primary plus live standbys whose
appliedLsn has reached the boundary. Every other caller, including the
settled final sweep, still compares all three. Log lines name members=.
Also logs bug-1854 (scheduler re-adds every configured resource on each
rejoin config push; ERROR noise, changed definitions ignored) for
follow-up.
10-round chaotic soak: PASS (divergence 0, job loss 0, SIMVERIFY 9 files,
0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The recorder encoded each controller event batch after
xce_core_push_events() had applied it, and applying moves payload fields
out of the events: a WORKERS_PUSH's resvals (core_apply_worker_resvals
MOVE_PTR), a JOBS_PUSH's allocs, a JOB_RUSAGE's rusages, and a
CONFIG_PUSH's shared resvals (sched_apply_pending_config recorded after
sched_init_config). Recordings silently lost exactly those fields -- a
worker's resval names were written with NULL values, so --sim-verify
replayed "no slots" and turned 3 live DISPATCHes into PENDs.
driver.c now records the batch before the push; the periodic in-stream
SnapshotEntry moves out of sched_record_write_events() into new
sched_record_checkpoint_after_apply(), called after the push so a
checkpoint still reflects the events written ahead of it. The config
push is likewise recorded before sched_init_config().
Regression test: test_recorded_workers_push_keeps_resval_values (RED:
recorded value NULL; GREEN after). Periodic-checkpoint tests updated to
the write-then-checkpoint contract. meson test 109/109; 25 live soak
recordings replay with 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_val_res() memcpy'd the xce_val_t into w->host.resvals, so the
"type"/"model" string resources aliased the interned xstrings owned by
w->host.attrs. rejoin_worker() drops the old attrs strings, then
apply_join_state() -> resync_arm_round() resolves the previous join's
still-outstanding round, firing the worker hooks before
add_hostattrs_resvals() re-seeds resvals; the apiserver hook's
collect_resval() strdup()ed the freed hostmodel. Live: a newly promoted
primary died ~8s after election during an HA soak.
A string resval now stores a private strdup (taken before the old copy is
freed), and the owning arrays free their entries: host.resvals gets
kill_resval (replacing the FIXME NULL killer) and c->sharedinst's
resval_obj_killer (previously a no-op) frees the string and the value --
both also leaked every xce_val_t before. Also closes the same alias for
worker-reported string resvals and config values.
Regression test: test_rejoin_worker_string_resvals_survive_outstanding_round
(RED reproduced the live ASAN stack; GREEN after). meson test 109/109.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follow-ups from the bug-1846 investigation.
bug-1847: config pushes never reached a .replay recording -- handler.c
applied them synchronously, bypassing the pending-update queue the
recorder writes from. With bug-1846's deferral they're now applied at a
round boundary on the driver thread, so sched_apply_pending_config()
writes a CONFIG_PUSH event via the existing sched_record_write_events()
(no-op when not recording), ahead of the round's other recorded events.
Test: test_config_push_deferred.c's test_applied_config_push_is_recorded
(RED: 0 events; GREEN: 1). Fidelity limit, pre-existing: replay applies
configPush through core_events.c's CONFIG_PUSH path, which doesn't re-run
groupmgr_load_configs()/plugin_config_update(), so a limits/groups change
still won't take effect in sim.
bug-1848: xce_sched_obj_t.hlimits was created with a NULL zarray killer
at all three constructors, so reset_sched_obj() and schedobj teardown
leaked the per-worker xlist_t match_job_host_limit_() callocs. New
kill_hlimits_list() (schedobj.c) frees the list, never its vres entries
(owned by the limit plugin); core_events.c and snapshot_load.c use
identical static local copies, since both avoid sharing helpers with
schedobj.c and several test binaries link them without it. Test:
test_limit_host_dedup.c's test_hlimits_zarray_releases_its_lists (RED:
killer NULL; GREEN).
bug-1849: every --sim run warned "failed to resolve job picker plugin ...
picker rebuild will be skipped". Benign: the snapshot is loaded before
the sim registers its picker, and sim_bootstrap_picker() replays the same
queue/job walk afterward. Downgraded to TRACE with an explanation.
Buglog also records bug-1850 (open): the ~100s reconnect churn behind
bug-1846 part 2 -- xcecli_retry_next() rotates/reconnects on every retry
(ETIMEDOUT after the 10s reply deadline, E_XCE_NOT_READY, redirects), so
a slow reply from a healthy primary forces a rejoin + full re-push.
Needs the live scheduler.log and a libxce spec.
Full scheduler suite 87/87 (asan). Replay of the captured file: same 184
mismatches, all attributable to the recording's pre-fix binary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.
Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.
Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.
Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
-- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
returned NULL right after the push), GREEN (session walks the remaining
jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
#include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).
Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
match_job_host_limit_() (plugins/limit.c) runs on every per-host
allocation attempt via limit_prepare_allocation(), and unconditionally
xlist_store()d the matched vres into obj->hlimits[worker->idx]. That list
is only rebuilt by limit_job_election() on a configver bump, so a job that
kept pending on the same host gained one more copy per attempt.
sched_fill_limit_requests() then staged every copy into
ctx->alloc_req.host, and seqjob's make_host_alloc() built a host alloc
carrying all of them.
Found via temporary per-decision --sim-verify diagnostics (reverted, not
committed) against scheduler-replay-20260923-085356.replay: pc13v host
allocs held mem + slots + ~70 identical limit entries (72) instead of 3.
After the fix the replay side is always 3 entries, and the LIVE recording
itself still shows the inflation (72/81/96/114/138/177 entries, resetting
only on a config push) -- so this is a production bug, not sim-only.
make_job_account() creates one res_account per entry and
sched_update_res_accounts() charges each, so the host limit was charged
(and later released) N-fold.
Fix: hlimits_contains_() pointer-identity check (vres is interned per name
in privdata->instances); a matched vres is stored once per (job, worker).
Test: test_limit_host_dedup.c (sched_limit_host_dedup_test) -- five
plugin_prepare_allocation() attempts for one job on one worker must leave
hlimits[worker]->count == 1 and one staged copy of the vres in
alloc_req.host. RED before the fix (Expected 1 Was 2), GREEN after; full
scheduler suite 86/86 (asan).
Still open (bug-1846 part 2, logged in buglog): 9 live-recorded rounds end
after 11-12 decisions while replay processes the whole candidate list; not
root-caused, needs the live scheduler.log or a fresh capture.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/xce-audit-invariants scheduler dimension-1/4 audit, prompted by a fresh
user-captured .replay file that still showed massive --sim-verify mismatch
volume after bug-1843's fix. Root-caused via temporary diagnostic
instrumentation against the real capture (reverted, never committed) rather
than trusting the surface symptom.
bug-1844: sim_verify_alloc_eq() (sim_verify.c) compared `instid`, a bare
per-process monotonic counter (scheduler.c's sched_get_id(), core->nextid)
stamped at allocation-build time -- not semantic content. The original
recording's counter and a fresh replay's counter have no reason to reach
the same value for "the same" dispatch. Live-verified: every DISPATCH
mismatch showed identical value/duration/resname/reserve, differing only
in instid (e.g. 826 vs 1). Dropped from the comparison, same treatment as
bug-1842's `from` exclusion.
bug-1845: sim_verify_job_preason_eq() required PEND decisions' full
host/resource reason-detail lists to match. core.h documents "simulation
isolation" as deliberate design: sim/replay never registers
core->unsatisfiable_lookup_cb, so it can never hit the unsatisfiable-cache
main.c's normal-mode branch registers. A live cache hit
(sched_pend_from_cached_reason_()) records only main_reason with zero
detail; replay always falls through to the full per-host computation and
always produces detail. Same semantic outcome (PEND, same main_reason) at
two architecturally-legitimate detail levels. Relaxed the comparator to
main_reason only. Separately verified the cache itself is sound (key is
safe within a round since capacity only decreases and the cache clears
every round) -- ruling it out as the cause of a second, still-open
divergence (bug-1846: one job genuinely dispatches one round early in
replay vs. the original recording -- a suspected resource-accounting
parity gap, not yet root-caused, logged for follow-up).
bug-1772 (previously deferred, incorrectly tagged "fixed"): re-found
independently by a dimension-4 snapshot-field-completeness audit subagent.
snapshot_build_primary_schedobj() (snapshot_load.c) never computed
resreq_fp, unlike its declared model core_build_primary_schedobj()
(core_events.c) -- every snapshot-restored job got a zeroed fingerprint,
which alloc_cache.c's cache key folds in directly alongside
user/queue/project, so two distinct restored jobs sharing those three
fields could silently share an incorrect unsatisfiable-cache verdict.
Added the same 4-field computation the live path uses.
TDD throughout: each fix has a dedicated regression test, and each was
verified via genuine red (revert fix, confirm the new test fails) then
green (restore, full 85/85 scheduler suite). Also ran three parallel
dimension audits (core I/O isolation, picker/allocator jbuf rendezvous) --
both clean, no violations found.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sched_record_session_add_decision() (recording_session.c) grows its
per-round decisions array via realloc(), doubling from 8. On a failed
growth it silently dropped the current decision and every one after it
that round, without marking the round incomplete -- finalize() then
wrote the short array as an ordinary, apparently-complete ReplaySession.
--sim-verify's replay comparison had no way to distinguish that from a
genuinely smaller live round, producing spurious "expected: <none>"
mismatches for jobs that were, in reality, correctly decided live and
simply never recorded. Live scheduling itself was unaffected either
way -- only the .replay file's own record of that one round lost its
tail.
This refutes the leading theory from the prior investigation (replay
seeing a job's first appearance "one round early" via coarser event
batching in sim_loop.c) -- confirmed via targeted diagnostic
instrumentation against the original captured .replay file: replay
deterministically reproduces the full candidate set every round;
the recording itself has the one-round gap.
Fix: track truncation on the session ctx; finalize() now refuses to
write an incomplete session and fails loudly through the same
sched_record_abort_on_error() path every other recording failure in
this subsystem already uses.
TDD: added a regression test forcing a real, deterministic realloc()
failure via -Wl,--wrap=realloc (scoped to this one test binary --
no prior malloc/realloc fault-injection harness existed in this
codebase). Verified genuine red -> green: the test fails without the
fix, passes with it. Full 85/85 scheduler suite green.
Note: this prevents future recordings from losing decisions this way;
it cannot retroactively repair an already-captured .replay file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- .wolf/anatomy-index.json, anatomy.md, memory.md: OpenWolf hook-maintained
project index and session log, refreshed by this session's activity.
- .kiro/specs/controller-peers-env-fallback/: spec scaffold (brief,
requirements, design, research) from prior spec-init work.
- issues/new_requirements.md: scheduler simulator enhancement backlog
(info-command improvements, pend-reason detail, decision tracing).
Excluded as non-persistent: issues/Issues.md (empty placeholder),
src/lib/testsuite/test_xcectx_reconnect_race.c (self-documented as an
ASAN stress reproduction only, not a committed regression test, and not
wired into meson.build), scheduler-replay-20260922-095207.replay
(4MB binary debug capture, no .replay files are tracked in this repo).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sim_verify_alloc_eq() compared xce_alloc_t.from, which make_alloc()
(xcealloc.c) stamps with the real OS wall clock (time(NULL)), not
core.c's virtualized sim clock. The recorded (expected) value is
whatever real second the original run happened to execute in; the
freshly-rebuilt (actual) value is whatever real second the replay
executes in -- two unrelated real-world timestamps, so every
otherwise-correct DISPATCH decision failed verification whenever a
replay ran even one wall-clock second away from its own recording.
Drop `from` from the comparison -- it's reservation bookkeeping, not
part of a decision's semantic identity. Verified against a real
captured .replay file: 120/120 false DISPATCH mismatches -> 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker-rejoin-resync sets XCE_WORKER_RECONCILING and pushes it to the
scheduler, which correctly freezes that worker's resInUse mid-reconciliation
(bug-1799/1801). But resync_resolve_round() only cleared the bit on the
controller's own worker_t, never notifying the scheduler -- so once a
rejoin round resolved, the scheduler's copy stayed RECONCILING forever,
permanently freezing that worker's resource accounting even after every
job on it finished. Live-observed via a replay capture: a 22-slot worker
stuck at used=22.00 for the rest of the run with zero RUN jobs left.
Added persist_notify_worker_status_updated() (mirrors bug-1679's job-side
persist_notify_job_updated_already_persisted()) and call it from
resync_resolve_round() so HOOK_DETAIL_WORKER_UPDATE fires without a
spurious WAL append. TDD: new spy-hook regression test, RED confirmed
against the unfixed code, GREEN after the fix. Full controller_unit_test
suite: 1220 Tests, 0 Failures, 0 Ignored.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live 3-node HA soak crashed xce_controller: SIGSEGV in
peer_ctrl_msg_handler_inline (ha.c:7035) dereferencing a NULL
ep->data. gdb analysis of the actual core dump showed the endpoint
itself fully alive (refcnt=191964, tearing_down=0) while ep->data had
been synchronously freed and nulled by standby_ep_err_handler_primary,
the endpoint's err_handler, with zero coordination with the vtask pool
that was concurrently dispatching a peer-control message for the same
endpoint (g_drive_async_peerctrl_dispatch_enabled defaults ON).
The sibling handlers (vote_ep_err_handler, pre_vote_ep_err_handler)
already avoid this exact hazard, with an explicit comment: "Do NOT
free vctx here -- peer_ctrl_msg_handler may still reference ep->data."
standby_ep_err_handler_primary was never updated to the same
discipline when the vtask-pool dispatch machinery (bug-1197/bug-1298)
was introduced.
Fix: route standby_ep_err_handler_primary through the same
already-tested vote_ep_teardown_async() deferred, per-endpoint-FIFO-
serialized teardown that vote-endpoint demotion already uses, plus an
idempotency guard (endpoint_is_tearing_down) since the old "ep->data
== NULL" reentry check no longer applies once the free is deferred.
Also fixes a second-order double-unref this surfaced in
on_standby_connected's stale-duplicate eviction path, caught by ASAN.
TDD: two new tests prove the teardown is deferred until an in-flight
same-ep vtask completes, and that a repeated error callback on the
same endpoint is a safe no-op. Full controller suite: 1219/1219, zero
regressions.
Also logs bug-1840 (a separate scheduler heap-corruption crash from
the same soak run) as not-fixed pending an ASAN reproduction -- a
non-instrumented core dump cannot localize glibc's malloc_consolidate
abort back to the actual out-of-bounds write.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sim_reconcile_unaccounted_jobs() (sim_loop.c) was calling
picker->picker_iface->on_dispatch() directly from the sim driver thread,
violating plugin.h's documented "allocator thread only" contract for
Post-allocation hooks. Dormant in practice only because builtin_fcfs.c
happens to lock internally around every hook -- a defensive measure the
contract text doesn't actually require of picker plugins.
Since scheduler-sim-live-allocation, sim mode spawns the same real
sched_job_picker()/sched_allocator() thread pair normal mode uses, so a
real allocator thread now exists to route through. Added a
reconcile_req/reconcile_ack cbuf pair on xce_sched_core_t and a reserved
XCE_SCHED_RECONCILE_WAKE sentinel pushed through jbuf to wake the
allocator thread without touching jbuf's pinned capacity-1/item-type
invariant (Req 5 AC 7). sched_allocator() services the request and calls
on_dispatch() itself; sim_reconcile_invoke_on_dispatch() blocks on the ack.
Falls back to a direct call when no allocator thread is live for a given
core (standalone unit tests), which is safe by construction.
TDD: new test spawns a stand-in allocator-relay thread and asserts
on_dispatch() fires there, not on the caller's thread -- confirmed RED
against the old direct-call code, GREEN after. Four other test files with
hand-rolled sched_allocator() stubs needed the same relay branch (found by
an actual hang at meson's timeout). Full scheduler suite: 85/85, zero
regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:
- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
Reconciliation" backstop) force-applied any recorded DISPATCH decision
it found, with zero live capacity check. Added
sim_reconciled_alloc_fits_capacity(), reusing the same
sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
the live picker/allocator path already uses.
- bug-1838: even with that gate, reconciliation still preempted every
freshly-PEND job before the live picker/allocator got a turn each round
-- unnecessary now that scheduler-sim-live-allocation gave sim mode the
same real, capacity-checked picker/allocator threads normal mode uses.
Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
2026-09-11 widening whose own justification no longer holds), so a
freshly-pushed job now always flows through the same live, verified
decision path as normal mode. Reconciliation now only ever fires for a
job already RUN from before the replay's own observation window --
the one case structurally invisible to the live picker.
Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.
Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.
Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CtrlHelloRequest.key was declared `Data` in controller.capnp. This
project's capnp-c backend silently no-ops encoding of Data-typed fields
without zero-initializing the caller's local struct, leaving an
uninitialized capn_ptr that write_CtrlHelloRequest unconditionally wrote
to the wire -- SIGSEGV in capn_setp/copy_ptr. Hit live: ctrl-1 won its
Raft election and crashed calling ha_send_ctrlhello_to_all_standbys().
Same bug class as bug-1504/bug-1505, both already fixed by switching the
field from Data to List(UInt8); bug-1505's own writeup had flagged
CtrlHelloRequest.key as a known-but-unfixed instance. Applied the
identical fix here: Data -> List(UInt8) + keyLen companion in the schema,
uint8_t*/key_len in typemap.h. The field is unused in production (always
NULL), so no consumer code changed. Added a round-trip regression test
covering both the NULL-key production shape and real key content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
update_limit_res_value() computed a percent-based limit's capacity from
the resource's own live, fluctuating value (e.g. "mem"), letting the
enforced cap drift with unrelated memory pressure instead of staying
pinned to a stable fraction of total capacity. instantiate_limit() now
resolves a static capacity-source resource once at instantiation time
(mem->maxmem, swap->maxswap, table-driven), stored on a new
xce_vres_t.percent_res field; a missing counterpart falls back to
today's behavior rather than failing instantiation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds an if/then JSON Schema (draft 2020-12) conditional constraint to
limits.schema.json's Limit definition: when percent is true, value
must be a number in [0.0, 1.0]. This is the class of authoring mistake
bug-1831/bug-1832 fixed manually (night_mem_pct: value: 50 instead of
0.5) -- now caught automatically at real `xceadmin config build` time
(the actual cluster.bin generation path deploy-ha.sh uses), not merely
by a unit test.
Empirically verified against the real jsonschema library this repo
uses: a violation surfaces with the exact same file/json_pointer/
constraint contract validate_yaml()'s existing ValidationError already
provides, so zero changes were needed to its error-reporting code.
While implementing this, the new constraint caught an identical
pre-existing defect in xceadmin's own checked-in "full" golden test
fixture (fixtures/full/configs/limits.yaml, night_mem_pct: value: 50)
-- fixed the same way, and regenerated the checked-in expected.bin
golden artifact to match the corrected input.
Added 3 tests to test_validation.py: reject an out-of-range percent
value, accept an in-range one, and confirm a non-percent value is
never range-checked regardless of magnitude.
Build gotcha discovered and documented (cerebrum.md): this repo has
two copies of every bundled schema -- src/configs/schema/ (canonical)
and src/xceadmin/src/xcecommon/schema_files/json/ (a gitignored local
mirror meson-build.sh always overwrites from the canonical copy on
every build). Only the canonical copy needed a source-controlled edit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
fmt_limit()/fmt_limit_units() formatted a percent-based limit's value
as a bare snprintf(..., "%f%%", limit->val.val) -- echoing the raw
stored value with a literal "%" appended. But xce_limit_val_t.val is
genuinely a 0-1 fraction when percent is set (confirmed end-to-end:
no /100 or *100 scaling exists anywhere in the config->capnp->runtime
pipeline; confirmed authoritative by the original hand-written C
literal this codebase's oldest percent-based limit was migrated from,
.val = 0.2 for a documented "20% mem cap"). A correctly-configured 20%
limit therefore displayed as "0.200000%" instead of "20.000000%" --
misleading, though technically an honest echo of the stored value.
Fixed by multiplying by 100.0 before formatting. Display-only --
update_limit_res_value()'s actual capacity computation was already
correct and is untouched. fmt_limit() (a thin XCE_UNITS_RAW wrapper)
inherits this fix for its own percent-branch output too, a deliberate,
intentional change from its pre-fix behavior for percent-based limits
specifically (its 5 pre-existing direct tests only assert rc > 0,
never the exact percent string, so this is safe).
Updated test_limit.c's percent-mode test fixture from an unrealistic
.val=80.0 to a realistic .val=0.8 (assertion text unchanged, since
0.8*100=80 either way) and test_repl_limit_table.c's own percent test
to expect its shared fixture's arbitrary .val=50.0 constant correctly
rendering as "5000.000000%". RED confirmed via the raw Unity testlog's
:FAIL: lines (the aggregate meson summary masks single-test failures,
a known harness quirk in this project); GREEN confirmed after, both
lib and scheduler suites, 0 Unity FAILs.
Discovered during live manual verification of scheduler-sim-diagnostics
Requirement 6 against a real deployed cluster.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
User's own independent retest against a real deployed cluster produced
a fully populated, correctly human-formatted mem-based limit usage
line (limits2: accounted=18022400.000000 (17.19 MB)
capacity=4635344384.000000 (4.32 GB)) -- directly contradicting the
original entry's "never populates" framing, which was based on a
single short-lived agent-run live test.
Downgraded from a confirmed defect to an unreproduced, timing/
session-shape-dependent observation. The original single-run finding
was real and remains unexplained, but a categorical "mem-based limits
don't populate usage via replay" claim is now known to be false.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered during live manual verification of scheduler-sim-diagnostics
Requirement 6 (human-readable info limit formatting) against a real
deployed cluster with a real captured replay. A mem-based limit
(limits2/ug1_global_mem/night_mem_pct) never shows a live usage
instance even when its matching job is genuinely dispatched with an
intact rusage[mem=...] reservation -- while a slots-based limit
(limit1) in the same replay correctly populates one. Suspected timing/
ordering gap between WORKERS_PUSH replay-event application and
match_job_global_limit_()/match_job_host_limit_()'s resmgr lookup, not
confirmed. Out of Requirement 6's boundary (display-formatting only,
proven correct independently) and out of bug-1826's already-closed
scope (matching invocation itself, not this resource-type-specific
timing gap). Needs its own dedicated investigation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
core_clock.c joined the mechanically-enforced isolation archive
(sched_core_isolation_lib) and its own doc comment claimed coverage by
the forbidden-headers/forbidden-symbols text scans, but was actually
absent from both -- only the nm(1) undefined-symbol layer covered it,
which cannot catch a forbidden #include resolving to no undefined
symbol (a macro-only header).
Added XCE_CORE_CLOCK_C_PATH (meson.build, mirroring the existing
per-file path macros) and extended both
test_core_never_includes_forbidden_headers() and
test_core_never_calls_forbidden_symbols() to scan core_clock.c. It
remains excluded only from the wall-clock scan, for its one sanctioned
clock_gettime() call.
TDD: confirmed RED by temporarily inserting a forbidden #include into
core_clock.c (84/85, correctly attributed failure), reverted
(byte-identical to HEAD), reconfirmed GREEN (85/85). No live violation
existed -- this closes a coverage gap in the mechanism itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Same pattern as bug-1492/1682/1611: tags claimed "fixed"/
"closed-by-scheduler-sim-live-allocation" while the fix narrative said
"NOT FIXED" -- and correctly so, that feature never touched
test_core_isolation.c. Removed the incorrect tags; entry remains open.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The dimension-1 audit (previous commit) reported bug-1611 as only
"partially fixed" -- but that assessment worked from bug-1611's own
stale narrative text without checking scheduler-sim-eager-reconciliation
(a separate spec, completed 2026-09-11, predating this session's audit).
That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger
from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in
the same round it is recorded, never deferred to a later confirmation
event -- closing bug-1611's original motivating scenario for good.
The "residual gap" my own first correction pass invented (an
already-RUN-at-snapshot-load job) is also not real: such a job's
allocation comes from the snapshot directly (confirmed field-complete
by this session's dimension-4 audit), so it's never a reconciliation
candidate in the first place.
Corrected bug-1611's buglog entry to FIXED and updated
scheduler-driver-equivalence.md's stale "Known violations" section
(still described both bug-1611 and bug-1613 as open) to reflect current
reality, with a pointer to re-verify rather than re-trust either framing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
/xce-audit-invariants scheduler (dimension 1, driver state-equivalence,
re-run after scheduler-sim-live-allocation) found plugin_end_session()
still checked `plugin->begin_session_fn != NULL` before invoking
`plugin->end_session_fn(plugin)` -- a copy-paste of
plugin_begin_session()'s own (correct) guard. Two failure modes: a
plugin registering only end_session_fn silently never has it invoked;
a plugin registering only begin_session_fn crashes via a NULL
function-pointer call. Currently inert in production (no live plugin
hits either shape) but a real latent defect in exactly the
session-boundary code the driver-equivalence invariant governs.
This exact defect had been independently filed twice: bug-1492
(2026-08-04, scheduler-core task 5.2's review) and bug-1682
(2026-08-24, found adjacent to bug-1681's fix) -- neither investigation
cross-referenced the other. Both entries also carried a "fixed" tag
inconsistent with their own "not yet fixed" narrative text, caught by
this audit's independent code+text cross-check.
Fixed the one-line guard; added test_end_session_invokes_end_session_fn_
when_only_it_is_set() (test_plugin_hook_dispatch.c) proving the silent
non-invocation, confirmed RED (84/85) against the buggy guard and GREEN
(85/85) after the fix. Updated both buglog entries to FIXED with
cross-references, and corrected bug-1611's stale narrative to reflect
its narrowing (not elimination) by scheduler-sim-live-allocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds the feature's concrete original acceptance test: a genuine
capnp-encoded .replay (real ReplayHeader/SnapshotEntry/Conf4Sched with
one global and one host-scoped limit), driven end-to-end through the
real, unmodified sched_sim_driver_run(). The dispatched job's usage
registers against both limits via limit_get_configured_limits()/
limit_get_usage_instances() -- the same accessors info limit reads --
proving real, non-zero usage where today's REPL previously showed none.
RED-phase evidence: reverting task 4.3's 4 files to their pre-fix
commit reproduces the exact predicted failure (host-scoped usage
missing, global usage intact); restoring to HEAD reproduces GREEN.
Closes bug-1826 (the bug this whole feature was built to fix) --
buglog entry updated from NOT FIXED to FULLY FIXED, citing this test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Fully closes bug-1829's own tracked follow-up from task 4.1: retypes
seqjob_allocator.c/parjob_allocator.c's init_fn AND runtime hooks
(prepare_allocation_hook, job_alloc_fn) to read the plugin's own
honest plugin->core field (task 1.2) instead of deriving an
xce_scheduler_t* from plugin->sched -- mirroring task 2.3's limit.c
migration exactly. The gap was broader than initially scoped: not
just init_fn, but every runtime hook independently derived sched from
plugin->sched on each allocation attempt.
Adds seqjob_allocator_new_plugin()/parjob_allocator_new_plugin()
factories (plugin.h, mirroring limit_new_plugin()), letting
sim_alloc.c's bootstrap functions register both plugins DIRECTLY
against &(sim->core) via the existing plugin_register_core() -- the
same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already
use. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the
plugin-registry/fan-out path entirely; plugin_prepare_allocation()'s
fan-out during a live seqjob/parjob dispatch now genuinely reaches
sim->core.plugins, the same registry the limit plugin is registered
in.
One narrow, necessary exception remains, independently verified
confined: my_alloc() still needs a small xce_scheduler_t wrapper
(only .core.resmgr aliased to sim->core) to satisfy sched_make_allocs()
(scheduler.c, out of this task's boundary, unmodified) -- set on
plugin->sched only AFTER plugin_register_core() has run, never read by
the plugin registry or any prepare_allocation_hook fan-out.
TDD: new test_sim_alloc_limit_fanout.c drives a real live dispatch
(sched_job_picker()/sched_allocator() thread pair) with a host-scoped
limit registered directly against sim->core, proving a real usage
instance is created -- isolated from both reconciliation and
global-limit matching. RED: dispatch succeeded but the usage-instance
iterator was empty. GREEN: 82/82, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Replaces sim_picker_thread_fn() (the always-PEND placeholder) with the
real sched_job_picker()/sched_allocator() threads (task 3.1), spawned
against &(sim->core) in sched_sim_driver_run(). Simulation mode now
genuinely uses the jbuf capacity-1 rendezvous for the first time
(confirmed still exactly one producer/one consumer). Adds
sched_election.c to sched_sim_isolation_lib.
Fixes bug-1829 (blocking, only reachable once sim_allocator() actually
called sched_make_allocation() for the first time): core->seqjob_
allocator/parjob_allocator were never bootstrapped for simulation mode
at all. Fixed via new sim_alloc.c (sim_bootstrap_seqjob_allocator()/
sim_bootstrap_parjob_allocator()), mirroring the sim_picker.c/
sim_limit.c precedent.
That fix's own workaround -- a throwaway xce_scheduler_t wrapper
satisfying seqjob/parjob's init_fn -- has a documented, narrow side
effect: prepare_allocation_hook fan-out during a live seqjob/parjob
dispatch attempt reaches the wrapper's own separate plugin registry,
not sim->core.plugins, so a plugin registered only against the real
sim core (the limit plugin) isn't reached for host-scoped matching
during a live dispatch yet. Documented in design.md's new "Known Seam
Gap" section; task 4.3 added to tasks.md to close it before task 5.2's
acceptance test can pass for the host-scoped case.
TDD: new end-to-end test drives a real .replay with an empty decisions
array (ruling out reconciliation) through the real thread pair,
proving a genuine DISPATCH from live resource availability. 25
pre-existing white-box sim/repl test files given link-satisfying
stubs reproducing the OLD always-PEND contract unchanged -- their own
assertions are untouched (task 5.3's job to reconcile deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Migrates match_job_global_limit_()/limit_job_election()/
match_job_host_limit_() (plugins/limit.c) to read plugin->core (added
task 1.2) instead of deriving xce_scheduler_t* from plugin->sched,
which stays NULL for a plugin registered via plugin_register_core()
(simulation mode). Removes the dead ctx.sched write flagged during
task 2.1's review -- verified this ctx never has .sched read anywhere
downstream.
Fixes bug-1828 (found while writing this task's own TDD test for a
previously untested code path): match_job_host_limit_() had the same
key_find()-negative-rc-treated-as-fatal defect bug-1786 already fixed
in its global-limit sibling, but never applied here -- meaning
host-scoped resource limits were never actually enforced, in any
mode, until this fix (not merely a display gap; the limit's vres was
never added to obj->hlimits, so sched_fill_limit_requests() always
saw an empty list for it).
TDD: new tests drive the real hook-registered entry points against a
plugin_register_core()-registered plugin, asserting real usage
instances via limit_get_usage_instances() for both global and
host-scoped limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
info limit was correctly implemented and tested but inert against a real
replay: register_limit()'s only call site was normal-mode sched_register_
plugins(); sim mode's own plugin bootstrap (sim_bootstrap_picker()) never
registered the limit plugin. The snapshot format already carries the
original session's limit config (sim->core.config, a borrowed reference set
by xce_sched_core_load_snapshot()) -- nothing needed to change in the
recording/replay format.
New sim_bootstrap_limit_plugin() (sim_limit.c, mirroring sim_picker.c's
structure) registers the plugin via plugin_register_core() and seeds it via
a new limit_seed_config(plugin, xce_sched_core_t* core, config) entry point,
wired into sim_loop.c alongside the existing sim_bootstrap_picker() call.
limit_new_plugin() extracted from register_limit() as a factory, mirroring
builtin_fcfs_new_plugin()'s established pattern; register_limit() itself is
behavior-preserving.
First attempt set plugin->sched to a disguised xce_sched_sim_driver_t*,
relying on it sharing xce_scheduler_t's first-member layout to reuse the
existing config_hook path -- rejected on review: this repeats the exact
cross-cast pattern that already corrupted memory once in this codebase (see
core.h's xce_core_cache_clear_cb_t history). Fixed by making
limit_seed_config() take xce_sched_core_t* as an honest parameter instead,
with limit_update_config() (normal mode) and the new sim path both
delegating to a shared limit_seed_config_impl() -- no cast, no aliasing risk,
independently verified by two review rounds reading the actual struct
layouts and call chains.
Also added sim_limit.c to sched_sim_isolation_lib so Req 5 AC 17's
controller-I/O-isolation check covers it (an initial gap, closed in the same
remediation round). Two ripple waves of the same established link-stub
pattern applied to the ~24 white-box test files that #include sim_loop.c.
info limit now shows real configured-limit data against a genuine replay.
Live usage will still correctly show "no active usage instance yet" until a
separate, larger fix (logged as bug-1826) wires limit *matching* into the
sim decision path -- deliberately out of this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified while closing out scheduler-sim-
diagnostics task 5: register_limit() has exactly one call site in the whole
codebase (normal-mode sched_register_plugins()); sim mode's own plugin
bootstrap (sim_bootstrap_picker()) only registers the picker plugin. The
info limit command built across this spec's 5 tasks is correctly
implemented and thoroughly tested but currently inert for a real operator
running xce_scheduler -i/-s <replay> -- it will always report "no limits
configured" regardless of the original recording's actual configuration.
Not fixed -- out of task 5's test-only boundary; recommended as the next
piece of work for this open-ended spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified (implementer + reviewer, both reading
src/lib/zarray.c directly) while implementing scheduler-sim-diagnostics task
3.1. Not fixed -- out of that task's scope; its own new code avoids the
mistake rather than touching the pre-existing function.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1820's own fix assumed sudo always forks exactly one direct child
that IS the real target, and looked for it via `pgrep -P $PID` (direct
children only). On a real host, sudo commonly interposes an additional
"monitor" process between the invoked sudo and the actual target
(sudo -> monitor -> target, three processes, not two -- standard
sudo 1.8.7+ architecture for signal/exit-status relay) -- a shape this
sandbox's own simpler sudo build doesn't reproduce, which is why
bug-1820 verified clean here but still left the real worker orphaned
on the reporting user's box. `pgrep -P` found and killed the monitor
(a direct child) while the actual worker (the monitor's own child, not
a direct child of the pidfile'd PID) survived untouched.
Replaces the parent/child tree-walking approach entirely with a
depth-independent identity-based re-scan: both generated scripts now
search system-wide (pgrep -f) for any still-running process whose own
cmdline matches the worker's distinguishing identity (-n $NAME plus
this deploy's -l $TOP/logs -- the same two args start-worker.sh's own
"already running" guard checks) and kill every match directly, however
many hops deep it landed. No longer needs to know or guess sudo's
internal process-tree depth.
Live-verified: regression-checked against this sandbox's own real
(2-level) sudo deploy (still clean), and built + killed a synthetic
3-level process tree modeled on the user's exact live report,
confirming the deepest process is found directly and the intermediate
hop then exits on its own once its child is gone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xceadmin's --controller option (config push, and scheduler record
start/stop/status) accepted only a single host:port. Passing every
configured controller as a comma-separated list -- the natural way to
query an HA cluster without knowing which node is alive/primary --
mis-parsed the whole joined string as one bogus hostname and failed
with a DNS-resolution error.
Adds xceadmin.discovery.resolve_controllers() (comma-split candidate
parsing) and follow_redirects_any() (tries each candidate in order,
skipping ones that are unreachable at the network level -- connection
refused, DNS failure, or a discovery timeout -- mirroring xcecli/
xce_test's own client-side master-rotation failover). A candidate that
IS reachable but returns a genuine protocol-level failure is not
retried against a different node. push.py and scheduler.py's own
duplicated single-host _split_controller helpers are removed in favor
of this shared implementation.
Also fixes a related gap bug-1821's own live verification surfaced:
right after a primary is killed, a standby doesn't start a new
election until its own heartbeat timeout elapses (~10s), so it keeps
redirecting every candidate to the now-dead old primary -- exhausting
the whole candidate list and raising NoPrimaryError, which
_send_admin_request()'s existing transient-retry loop (bug-1713/1715)
never caught. That loop now retries on NoPrimaryError too, using its
own (longer) budget sized for the heartbeat-timeout-plus-election
window rather than the RPC-level "already electing" budget the
original retry logic was tuned for.
TDD RED->GREEN, 304/304 tests passing. Live-verified end to end: a
comma-list correctly skips a dead first candidate and reaches a live
one on a healthy cluster, and correctly rides out a real primary kill
and re-election (retry loop visibly cycling ~15s) to reach the newly-
elected primary rather than failing immediately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
start-worker.sh's real-worker branch writes the pidfile via `echo $$`
then `exec sudo "$WORKER_BIN" ...` -- exec replaces the pidfile-writing
shell with sudo in place (same PID), so the pidfile records sudo's own
PID, not the actual xcecd daemon's. sudo forks a separate, root-owned
child to run the real worker; kill -9 on sudo's PID kills only sudo
(SIGKILL can't be forwarded to a child), silently orphaning xcecd. The
existing "still alive after kill -9" warning never caught this either,
since it re-checked the same (already-dead) sudo PID.
Both generated stop-all.sh and cleanup.sh now detect a sudo-wrapped
pidfile entry (its /proc/$PID/cmdline starts with "sudo "), find its
child via pgrep -P, and kill that child with sudo first. The
still-alive check now also tests for the child's continued existence.
Live-verified end-to-end: deployed a 3-node cluster with a real
(XCE_REAL_WORKER=1) worker, confirmed the sudo-parent/root-child PID
pair, ran the fixed stop-all.sh and cleanup.sh and confirmed zero
processes remain after each, with the new "is a sudo wrapper" line
printed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Found live during task 8.5's optional catching-up-observation attempt:
a killed-then-reconnected standby can end up occupying a NEW
replicator_t.standbys[] slot while its OLD slot (host/port still
populated, ep == NULL) also matches the same configured peer.
ctrlstatus_fill's primary-side peer-status loop stopped at the first
host:port match, so it permanently reported that peer dead even
minutes after it had genuinely reconnected and fully caught up --
violating Requirement 13.3's "true, currently-known connection status"
guarantee.
Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot
with ep != NULL over a slot with ep == NULL; fall back to dead only
when no matching slot is live. Read-only, lock scope unchanged from
task 8.2.
TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced
by the reviewer (exact RED replay) plus two independent live
kill/backlog/restart/poll cycles on separate fresh clusters, both
confirming the previously-stuck standby now converges to ok.
design.md's Locked Decision 3 corrected to document the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce-build.sh previously left --jobs unset with no fallback -- a bare/
direct invocation (CI, a manual `bash xce-build.sh`, anything bypassing
the /xce-build skill) got meson/ninja/go's own unthrottled
auto-parallelism (~nproc+2), risking starvation on this sandbox's
routinely-concurrent build/test sessions.
The script now computes the same live default the /xce-build skill has
always computed by hand -- nproc, available memory, and load average
read fresh at invocation, with a per-profile memory budget (1536 MB/job
asan, 1024 MB/job debug/release) and process-factor discount (0.5 asan/
debug, 0.7 release) for the compiler driver's forked cc1/as children --
whenever --jobs isn't passed explicitly. An explicit --jobs N still
always wins; skill-driven builds are unaffected since the skill already
passes its own computed value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.
The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.
test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
pthread_cond_wait() reacquires its mutex before honoring a cancellation,
but mylock.c installed no cleanup handler to release it again -- a
thread cancelled mid-wait terminated holding the lock forever. vtaskmgr's
pthread_cancel()-based teardown could land there from three angles
(dispatcher cancelled while polling the shared queue, or a vcpu racing
back into it after its own shutdown signal), permanently deadlocking any
other thread that touched the same queue and hanging vtaskmgr_fini()'s
pthread_join() indefinitely.
Reproduced live as xce_test hanging on Ctrl-D exit once a connection
context was entered; gdb confirmed a vcpu thread blocked forever on a
mutex owned by an already-exited thread ID.
Fix: cbuf_put()/cbuf_get()/cbuf_timedget() now unlock via a
pthread_cleanup_push/pop handler around the wait, so a cancelled waiter
always releases the mutex. vcpu_idle()/vcpu_entry() also recognize the
teardown poison pill and exit immediately instead of needlessly
re-announcing into the already-dead shared queue.
New test_vtaskmgr_teardown.c reproduces the race deterministically
(self-pinned to one CPU, since true multi-core parallelism rarely hits
the window) and bounds vtaskmgr_fini() with a timed join so a
regression fails fast instead of hanging the suite. Logged as bug-1810.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce_core_push_events() WARNed on every JOBS_PUSH resync redelivery
(sched_add_job rc=-EEXIST), which is the expected outcome under
on_scheduler_join()'s full resync or the array-push dedup race
(bug-1734/bug-1714) and flooded operational logs. Only that case now
logs via a new DBG() macro (utils.h); any other add_rc (e.g. -ENOMEM)
still WARNs. DBG is named to avoid colliding with the build's own
-DDEBUG command-line define (debug/asan buildtypes).
Also collapsed hash_string()'s duplicated djb2 loop to delegate to
hash_block(), which already implements the same RUN_HASH loop.
Logged as bug-1809 in .wolf/buglog.json.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Every cmd_* REPL handler in test.c/join_worker_args.c reset flag parsing
with `optind = 1;` only. glibc's getopt_long only fully reinitializes its
internal scan state (nextchar/first_nonopt/last_nonopt) when optind==0, or
on the process's very first getopt_long() call ever -- optind=1 alone
leaves that state stale for every call after the first. Since xce_test's
REPL dispatches many independent cmd_*() calls per process, a command
whose parse ends on a short no-argument flag (`qctrl -a`, `hostctrl -o`)
left state that corrupted the NEXT unrelated command's parse badly enough
to misread its own name as an unrecognized flag. This was the actual
cause of harness_unit_test's 4 deterministic (not flaky -- 20/20 identical
repeat runs) test_test_cli.c failures, each burning its full 5s
confirmation-prompt timeout. Fixed by using optind=0 at all 21 reset
sites; confirmed via 3 isolated pty repros plus 10 consecutive clean
harness_unit_test runs (132/132, was 4 failures/run every time).
Also fixes a resource leak found while investigating: test_test_cli.c's
spawned xce_test sessions were only torn down by inline
clitest_kill_and_reap()/clitest_close() calls at the bottom of each test
function, which Unity's longjmp-based TEST_ASSERT abort skips entirely on
failure, leaking the child process and its pty/pipe fds. Added a static
session mirror (safe across the longjmp, unlike a stack pointer) reaped
from harness_test.c's tearDown(), which Unity guarantees runs after every
test regardless of outcome.
Logged as bug-1807/bug-1808 in .wolf/buglog.json, with the underlying
getopt/longjmp gotchas recorded in .wolf/cerebrum.md. Full ASAN
/xce-build harness also passes 5/5 with no sanitizer findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
