Adds a NULL-by-default lookup/populate callback seam on xce_sched_core_t,
mirroring cache_clear_cb's proven shape, so the allocation dispatch
chokepoint can consult the unsatisfiable-resreq cache without a
normal-mode-only field access in shared code. Adapters live in
alloc_cache.c and are registered only during normal-mode startup;
simulation mode leaves both callbacks NULL and always computes fresh.
Adds the scheduler-sim-live-allocation spec (requirements/design/
research/tasks) as the planning artifact this and subsequent tasks
implement against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
info limit was correctly implemented and tested but inert against a real
replay: register_limit()'s only call site was normal-mode sched_register_
plugins(); sim mode's own plugin bootstrap (sim_bootstrap_picker()) never
registered the limit plugin. The snapshot format already carries the
original session's limit config (sim->core.config, a borrowed reference set
by xce_sched_core_load_snapshot()) -- nothing needed to change in the
recording/replay format.
New sim_bootstrap_limit_plugin() (sim_limit.c, mirroring sim_picker.c's
structure) registers the plugin via plugin_register_core() and seeds it via
a new limit_seed_config(plugin, xce_sched_core_t* core, config) entry point,
wired into sim_loop.c alongside the existing sim_bootstrap_picker() call.
limit_new_plugin() extracted from register_limit() as a factory, mirroring
builtin_fcfs_new_plugin()'s established pattern; register_limit() itself is
behavior-preserving.
First attempt set plugin->sched to a disguised xce_sched_sim_driver_t*,
relying on it sharing xce_scheduler_t's first-member layout to reuse the
existing config_hook path -- rejected on review: this repeats the exact
cross-cast pattern that already corrupted memory once in this codebase (see
core.h's xce_core_cache_clear_cb_t history). Fixed by making
limit_seed_config() take xce_sched_core_t* as an honest parameter instead,
with limit_update_config() (normal mode) and the new sim path both
delegating to a shared limit_seed_config_impl() -- no cast, no aliasing risk,
independently verified by two review rounds reading the actual struct
layouts and call chains.
Also added sim_limit.c to sched_sim_isolation_lib so Req 5 AC 17's
controller-I/O-isolation check covers it (an initial gap, closed in the same
remediation round). Two ripple waves of the same established link-stub
pattern applied to the ~24 white-box test files that #include sim_loop.c.
info limit now shows real configured-limit data against a genuine replay.
Live usage will still correctly show "no active usage instance yet" until a
separate, larger fix (logged as bug-1826) wires limit *matching* into the
sim decision path -- deliberately out of this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Records the investigation and precise design before implementation: the
snapshot format already carries the original limit config (sim->core.config,
borrowed reference set by xce_sched_core_load_snapshot()), so the fix is
registration + a direct config_hook seed, mirroring sim_bootstrap_picker()'s
existing pattern exactly -- not a recording/replay format change. Also
documents the separate, deliberately out-of-scope finding that limit
*matching* during replay needs a further, larger fix (plugin_job_election()
is normal-mode-only today).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Tasks 2 and 3 are container-only major headers (per tasks-generation.md's
hierarchy rule); all their executable sub-tasks (2.1-2.3, 3.1-3.2) are
already complete. Bookkeeping only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified while closing out scheduler-sim-
diagnostics task 5: register_limit() has exactly one call site in the whole
codebase (normal-mode sched_register_plugins()); sim mode's own plugin
bootstrap (sim_bootstrap_picker()) only registers the picker plugin. The
info limit command built across this spec's 5 tasks is correctly
implemented and thoroughly tested but currently inert for a real operator
running xce_scheduler -i/-s <replay> -- it will always report "no limits
configured" regardless of the original recording's actual configuration.
Not fixed -- out of task 5's test-only boundary; recommended as the next
piece of work for this open-ended spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Four new tests in test_repl_limit_table.c close out Requirement 5:
- test_info_limit_never_mutates_global_limit_config_or_accounting_state /
..._host_scoped_... : snapshot a matched limit's config fields and its
vres capacity/in-use zarray entries (both pointer identity and value)
before and after repeated info limit calls; asserts byte-identical state
and byte-identical repeated stdout output.
- test_info_limit_on_empty_registry_is_repeatable_and_never_registers_plugin:
confirms info limit never causes the plugin to become spuriously
registered as a side effect of being queried.
- test_repl_step_continue_with_info_limit_interleaved_matches_baseline_decision_sequence:
a step/continue sequence run twice (once with info limit interleaved, once
without) produces an identical decision sequence.
Verified via real mutation testing: three targeted fake mutations were
temporarily applied to repl_info.c, confirmed each new test fails exactly as
predicted, then cleanly reverted (repl_info.c shows zero diff from HEAD).
Zero production code changed -- this is a test-only validation task. Full
14-binary regression sweep + 78/78 meson aggregate confirm zero impact on
any pre-existing info subcommand (Requirement 5.3).
This closes out scheduler-sim-diagnostics' first capability (limit
inspection, Requirements 1-5) end to end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_cmd_info() now recognizes "limit" as an info subcommand, routing to the
already-implemented and tested repl_info_limit() with the same ntok<3 usage
guard shape as the queue/worker/resource branches. Usage string and help
text updated to list it. repl_info_limit()'s own body is untouched -- this
is a pure, additive dispatch-wiring change. Two new tests drive it through
the real repl_dispatch() entry point (not a direct call) for both `info
limit <name>` and `info limit all`.
`info limit <name|all>` is now reachable from a live REPL session --
scheduler-sim-diagnostics' first capability is functionally complete.
Remaining: task 5 (determinism/regression validation).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_print_limit_usage_per_host() fans a single host-scoped xce_vres_t out
across every worker with a populated capacity entry at that vres's shared
resource index (w->resvals[idx]/w->resInUse[idx]), rather than constructing
any per-host vres object -- matching design.md's (xce_vres_t, worker) pair
cardinality model exactly. A worker with no populated capacity entry produces
no line at all (not a fabricated zero); a worker with capacity but no
charged usage shows accounted=0. Uses the corrected idx<0 + .caps-only
bounds-check pattern task 3.1 established, avoiding bug-1824's latent
imprecision. This closes out all of Requirement 3 (3.1-3.4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified (implementer + reviewer, both reading
src/lib/zarray.c directly) while implementing scheduler-sim-diagnostics task
3.1. Not fixed -- out of that task's scope; its own new code avoids the
mistake rather than touching the pre-existing function.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now displays accounted-usage/capacity for GLOBAL/SHARED
limit instances: iterates limit_get_usage_instances(), filters by
vres->limit == matched_limit, reads capacity from core.sharedInst[idx] and
accounted usage from core.sharedInUse[idx]. Existence-of-instance and
zero-usage are kept distinct per design.md: no matching instance (or one
with no populated capacity) prints "no active usage instance yet"; a
matched-but-never-charged instance correctly shows accounted=0 instead.
Self-caught during implementation and independently confirmed by review: a
negative or stale vres->idx (possible per add_vres()'s allocate-then-assign
pattern) is not safely rejected by zarray_get_ex()'s idx>=caps check alone
-- the new code adds an explicit idx<0 guard rather than copying
repl_info_resource_one()'s existing .count-based pre-check (which is itself
imprecise for a sparsely-populated zarray, a pre-existing issue left
out of scope here).
Calling limit_get_usage_instances() for the first time from repl_info.c
reproduced the same ~24-file link-stub ripple task 2.1 established for
limit_get_configured_limits() -- fixed with the identical pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now prints an explanatory note when a limit is implicitly
host-scoped (is_host_based_limit() > 0 due to a percent-based value on a
non-shared resource) but its own configuration doesn't show an explicit
per-host scope. The real discriminator turned out to be consumers[CONSUMER_
HOST]->per_consumer, not pointer-nullity as design.md's gloss said --
build_consumer() unconditionally allocates a consumer for all four
dimensions, so the struct field is never actually NULL in production. Fixed
using the same field is_host_based_limit() itself already falls back to;
verified via RED-phase reproduction that the literal design.md condition
never fires. design.md corrected to document the real condition.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now prints a matched limit's configured setting on both the
all and named-lookup success paths: fmt_limit() (existing libxce formatter,
unmodified) into a stack buffer, re-split and printed through a new
repl_print_limit_dump() helper matching this file's existing [sim]-prefix-
per-line convention. Inclusion and exclusion group membership both render
since fmt_group() already handles both, adopted as-is per design.md's
explicit reuse decision.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() (repl_info.c, not yet wired into repl_cmd_info() -- that's
task 4) resolves "all" vs a specific name against limit_get_configured_limits():
a NULL/empty handle on the all path reports "no limits configured" (Req 1.7);
a NULL handle or key_find() miss on the named path reports "no such limit
'<name>'" (Req 1.3) and never collapses into the all-only message. New
white-box test_repl_limit_table.c exercises both paths directly, including a
dedicated regression guard for the NULL-scoping rule.
Making repl_info.c genuinely call limit_get_configured_limits() broke the link
for ~24 other white-box test files that #include repl_info.c without also
linking plugins/limit.c -- fixed with the same link-satisfying-stub idiom this
file family already uses elsewhere (all return NULL, verified behavior-
preserving since none of these fixtures register the limit plugin).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
limit_get_configured_limits()/limit_get_usage_instances() cross the limit
plugin's private-state boundary (plugin->data, previously unreachable outside
limit.c) via key_find on the name-keyed plugin registry, returning NULL when
the plugin is absent/uninitialized and a raw iterable kvmgr handle otherwise.
Pure reads only -- no allocation, mutation, or hook triggering. Foundation for
the upcoming `info limit` REPL command (task 1 of 5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Requirement set 1 (info limit <name|all>) of the open-ended scheduler-sim-diagnostics
spec: requirements refined for effective vs. configured scope and exclusion-group
membership, design grounded in the actual limit-plugin/REPL source (accessor boundary,
GLOBAL/SHARED-vs-per-host read split, existence-vs-zero-usage distinction), and an
8-subtask implementation plan with full requirements traceability.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1820's own fix assumed sudo always forks exactly one direct child
that IS the real target, and looked for it via `pgrep -P $PID` (direct
children only). On a real host, sudo commonly interposes an additional
"monitor" process between the invoked sudo and the actual target
(sudo -> monitor -> target, three processes, not two -- standard
sudo 1.8.7+ architecture for signal/exit-status relay) -- a shape this
sandbox's own simpler sudo build doesn't reproduce, which is why
bug-1820 verified clean here but still left the real worker orphaned
on the reporting user's box. `pgrep -P` found and killed the monitor
(a direct child) while the actual worker (the monitor's own child, not
a direct child of the pidfile'd PID) survived untouched.
Replaces the parent/child tree-walking approach entirely with a
depth-independent identity-based re-scan: both generated scripts now
search system-wide (pgrep -f) for any still-running process whose own
cmdline matches the worker's distinguishing identity (-n $NAME plus
this deploy's -l $TOP/logs -- the same two args start-worker.sh's own
"already running" guard checks) and kill every match directly, however
many hops deep it landed. No longer needs to know or guess sudo's
internal process-tree depth.
Live-verified: regression-checked against this sandbox's own real
(2-level) sudo deploy (still clean), and built + killed a synthetic
3-level process tree modeled on the user's exact live report,
confirming the deepest process is found directly and the intermediate
hop then exits on its own once its child is gone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xceadmin's --controller option (config push, and scheduler record
start/stop/status) accepted only a single host:port. Passing every
configured controller as a comma-separated list -- the natural way to
query an HA cluster without knowing which node is alive/primary --
mis-parsed the whole joined string as one bogus hostname and failed
with a DNS-resolution error.
Adds xceadmin.discovery.resolve_controllers() (comma-split candidate
parsing) and follow_redirects_any() (tries each candidate in order,
skipping ones that are unreachable at the network level -- connection
refused, DNS failure, or a discovery timeout -- mirroring xcecli/
xce_test's own client-side master-rotation failover). A candidate that
IS reachable but returns a genuine protocol-level failure is not
retried against a different node. push.py and scheduler.py's own
duplicated single-host _split_controller helpers are removed in favor
of this shared implementation.
Also fixes a related gap bug-1821's own live verification surfaced:
right after a primary is killed, a standby doesn't start a new
election until its own heartbeat timeout elapses (~10s), so it keeps
redirecting every candidate to the now-dead old primary -- exhausting
the whole candidate list and raising NoPrimaryError, which
_send_admin_request()'s existing transient-retry loop (bug-1713/1715)
never caught. That loop now retries on NoPrimaryError too, using its
own (longer) budget sized for the heartbeat-timeout-plus-election
window rather than the RPC-level "already electing" budget the
original retry logic was tuned for.
TDD RED->GREEN, 304/304 tests passing. Live-verified end to end: a
comma-list correctly skips a dead first candidate and reaches a live
one on a healthy cluster, and correctly rides out a real primary kill
and re-election (retry loop visibly cycling ~15s) to reach the newly-
elected primary rather than failing immediately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
start-worker.sh's real-worker branch writes the pidfile via `echo $$`
then `exec sudo "$WORKER_BIN" ...` -- exec replaces the pidfile-writing
shell with sudo in place (same PID), so the pidfile records sudo's own
PID, not the actual xcecd daemon's. sudo forks a separate, root-owned
child to run the real worker; kill -9 on sudo's PID kills only sudo
(SIGKILL can't be forwarded to a child), silently orphaning xcecd. The
existing "still alive after kill -9" warning never caught this either,
since it re-checked the same (already-dead) sudo PID.
Both generated stop-all.sh and cleanup.sh now detect a sudo-wrapped
pidfile entry (its /proc/$PID/cmdline starts with "sudo "), find its
child via pgrep -P, and kill that child with sudo first. The
still-alive check now also tests for the child's continued existence.
Live-verified end-to-end: deployed a 3-node cluster with a real
(XCE_REAL_WORKER=1) worker, confirmed the sudo-parent/root-child PID
pair, ran the fixed stop-all.sh and cleanup.sh and confirmed zero
processes remain after each, with the new "is a sudo wrapper" line
printed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Found live during task 8.5's optional catching-up-observation attempt:
a killed-then-reconnected standby can end up occupying a NEW
replicator_t.standbys[] slot while its OLD slot (host/port still
populated, ep == NULL) also matches the same configured peer.
ctrlstatus_fill's primary-side peer-status loop stopped at the first
host:port match, so it permanently reported that peer dead even
minutes after it had genuinely reconnected and fully caught up --
violating Requirement 13.3's "true, currently-known connection status"
guarantee.
Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot
with ep != NULL over a slot with ep == NULL; fall back to dead only
when no matching slot is live. Read-only, lock scope unchanged from
task 8.2.
TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced
by the reviewer (exact RED replay) plus two independent live
kill/backlog/restart/poll cycles on separate fresh clusters, both
confirming the previously-stuck standby now converges to ok.
design.md's Locked Decision 3 corrected to document the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Two independent live 3-node-cluster runs both confirmed the full
Requirement 13 scenario (healthy all-ok, dead-on-kill, unknown-from-
standby-view) with zero bugs found. No source changes -- pure
validation task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds ctrlstatus_peer_status_str (mirrors ctrlstatus_role_str's
convention: 0=unknown/1=ok/2=catching-up/3=dead, safe default fallback
for any other code) and extends cmd_ctrlstatus's per-controller print
line to render it alongside each configured controller's existing
name:address, completing the CtrlPeer.status wire field added in task
8.1 and sourced in task 8.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Populates CtrlPeer.status (added in task 8.1) inside ctrlstatus_fill's
peer-building loop: self always reports ok; a primary responder sources
every other peer from the live replication connection registry
(replicator_t.standbys[], host+port matched, reusing
repl_active_standby_count's ep!=NULL && !in_catch_up predicate) --
dead/catching-up/ok; a non-primary responder reports unknown for every
non-self peer. A self-identity mismatch (pre-existing misconfiguration)
logs one WARN and falls through to the normal non-self sourcing rather
than forcing any peer to ok. Read-only: a bounded lock-protected scan,
no state mutation.
TDD: 3 new tests (primary-sourced-from-registry, standby-unknown-except-
self, self-mismatch-falls-through) plus an extended no-mutation
assertion; RED->GREEN independently reproduced by the reviewer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlPeer.status (UInt8, next free ordinal @3 after port) plus the
matching xce_ctrlpeer_t.status typemap field, verified field-for-field
against the regenerated codec with a genuine encode/decode round-trip
test. No sourcing/rendering logic yet (task 8.2/8.3) -- field defaults
to 0 (unknown) until wired.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Second design-review pass caught a factual error: the Constraints
section claimed ctrlstatus_fill "already takes ha->lock" for the
primary-identity read, citing the ORIGINAL Req 1-9 design text rather
than the as-built code -- after the ade0081 fix, that read takes no
lock at all. Corrected: this delta's replicator_t.lock acquisition is
the first lock ctrlstatus_fill has taken since that fix, not a repeat
of an existing in-function pattern (still safe, per the existing
repl_active_standby_count precedent elsewhere in the codebase).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds requirements, design, and research for showing each configured
controller's connection status (ok/catching-up/dead/unknown) in
ctrlstatus's peer list, with the same off-primary fidelity pattern
already established (primary reports the true picture; a standby
reports unknown for what it cannot itself observe).
Design review found and fixed one gap before implementation: no
defined fallback if the responder's own name doesn't match any
configured peer entry (a pre-existing misconfiguration class, not new
to this delta) -- now an explicit, logged (WARN) degradation instead
of a silently wrong status.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
/kiro-validate-impl: full suite green (controller 1210/0, lib 368/0,
harness 173/0), live 3-node smoke boot confirms bug-1818's fix holds,
12/12 requirement groups covered, cross-task integration traced at the
source level, dependency direction clean, zero blocked tasks. Prior
2026-07-17 validate_impl record preserved under prior_validate_impl.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Feature-level /kiro-validate-impl found the delta's File Structure Plan
omitted 5 as-built files task 7.3 actually needed (request.h/request.c
-- the real heartbeat_msg_t struct-builder -- plus 3 cascading
signature-conformance test files). Documents the as-built reality and
the already-fixed cmd_ctrlstatus gate text; also notes the pre-existing,
deliberate lack of a checked-in unit test for cmd_ctrlstatus's print
layer (the class of gap bug-1818 exploited) as a known, accepted scope
boundary for future readers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
phase -> implementation-complete; delta.impl_status records tasks
7.1-7.6 done, including the bug-1818 find-and-fix from live E2E.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Requirement 12 delta (primary election term/time) is now fully
implemented: tasks 7.1-7.6 all complete, including the live 3-node
failover validation that found and confirmed the fix for bug-1818.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Live 3-node E2E validation (task 7.6) found that cmd_ctrlstatus's
electedTerm/electedAt lines (task 7.5) gated on NULLSTR(primaryName),
which is unconditionally empty on every non-primary responder in this
build (ctrlstatus_fill only self-reports the primary's own name --
ha->primary_host is vestigial, a pre-existing Req 2 behavior). This made
electedTerm/electedAt print "unknown" on every standby, forever, even
though elected_at_ms was correctly, non-zero, and convergently populated
the whole time (term/committedLsn/appliedLsn, sourced the same
unconditional way, DID correctly display and converge on the same query).
Fix: gate on status->elected_at_ms == 0 instead -- the field stays
exactly 0 until a real promotion stamps it (either road) or a heartbeat
carries a learned value from an actual primary, unlike term (which can
legitimately bump from mere vote/candidacy activity before any primary
is elected). ctrlstatus_fill itself is untouched; this is purely a
cmd_ctrlstatus display-layer correction.
Also corrects design.md's "Design Delta -- Requirement 12" Locked
Decision 3 (and its dependent traceability/service-contract text) to
document this live-E2E finding, mirroring the spec's existing delta
convention (Req 10/11's own live-E2E correction).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
cmd_ctrlstatus (src/harness/test.c) now prints electedTerm/electedAt
right after the Primary: line, gated on the same NULLSTR(primaryName)
"unknown" check that line already uses. electedTerm prints the existing
term field verbatim (no new wire field, per design's Locked Decision 1);
electedAt reuses the existing ctrlstatus_format_time_ms formatter
already used for responderTime, falling back to "unknown" on a NULL
format result. No leakage of a stale term/elected_at_ms when no primary
is known.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
ctrlstatus_fill now unconditionally reads ha->last_known_elected_at_ms
into the reply's elected_at_ms field -- one atomic-acquire load, no
role-branch, mirroring how the handler already reports raw term
unconditionally. Serves both a self-stamped primary and a
heartbeat-learned standby from the same field. "Unknown" display
gating is deliberately left to the client command (task 7.5), not
this handler.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The primary echoes its own last_known_elected_at_ms on every outgoing
heartbeat (threaded through xce_ha_heartbeat_req's existing scalar-arg
plumbing, alongside term/committed_lsn/committed_config_tag); a standby
processing a received heartbeat (ha_consume_heartbeat_committed, the sole
production call site) stores it unconditionally into its own
last_known_elected_at_ms, reconcile-free -- no other HA field moves.
ha_observe_heartbeat_committed itself is untouched; the new store sits
next to its call site, not inside it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Stamps ha_state_t.last_known_elected_at_ms (CLOCK_REALTIME, epoch-ms) in
both ha_do_promote (non-raft road) and raft_cb_become_leader (raft road),
placed at each road's genuine, final commit point to CTRL_PRIMARY -- past
every early-return refusal path in raft_cb_become_leader (first-boot
phase2 failure, the Live Apply Resource Fence re-check, the promotion
eligibility gate, config-serve-fenced, upgrade-materialization-fenced).
Purely observational: does not change whether/when a promotion succeeds
or is refused. Three new tests (2 positive, 1 negative) prove placement:
a refused promotion leaves the field at its pre-attempt value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlStatus.electedAtMs@14 and HeartbeatMsg.electedAtMs@4 to the
capnp schema, their hand-authored typemap.h counterparts, and a new
ha_state_t.last_known_elected_at_ms field alongside last_known_committed_lsn.
Schema/state plumbing only -- nothing stamps or consumes these fields yet
(tasks 7.2-7.5). Codec round-trip tests confirm both new fields survive
encode/decode unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds requirements, design, research, and tasks for reporting the term
and wall-clock time the current primary was elected, including
standby-side learned fidelity (mirroring the Req 10 pattern), following
the existing controller-status spec's delta convention (Req 10/11).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
A user-reported core dump showed xce_scheduler crashing with SIGSEGV in
its allocator thread: zarray_get_ex dereferencing a NULL array->data
inside match_job_host_limit_() (plugins/limit.c), reached via schedobj.c's
reset_sched_obj().
Root cause: zarray_fini() freed array->data and set it to NULL, but left
array->caps/count/pos at their stale pre-fini values instead of resetting
to the safe (0,0,0) state zarray_init() establishes. xlist_fini() had the
identical gap. Both structs' own bounds checks (`idx >= caps`) treat a
nonzero caps as a guarantee that data is a valid allocation -- a
guarantee broken for any caller that reuses the same struct after
_fini() without re-init.
schedobj.c's reset_sched_obj() does exactly that: it tears down a job's
own PERSISTENT schedobj (reused for the job's whole lifetime, not a
disposable scratch object) whenever the cluster's config version has
advanced, with no matching re-init. Once a host-limit match had grown
hlimits' capacity, and the config version bumped again while that job
sat PEND, the next allocation attempt for the same job dereferenced the
torn array -- exactly matching the reported backtrace. This same crash
class had already caused at least one earlier, never-successfully-root-
caused crash near the same call site (2026-08-17).
Fixed at the library-primitive level rather than reset_sched_obj()'s
call site: zarray_fini()/xlist_fini() now reset caps/count/pos to 0,
matching what _init() establishes. This makes every existing
_fini()-then-reuse caller correct by construction, with zero changes
needed at any call site. Audited every non-test zarray_fini()/
xlist_fini() call site first -- none relies on caps/count surviving
_fini(), so this is a pure safety improvement, not a behavior change.
RED confirmed by temporarily disabling both resets: new tests failed
cleanly ("Expected 0 Was 4" / "Expected 0 Was 2", the stale pre-fini
caps). GREEN confirmed restored: full lib suite 314/314, scheduler
suite 77/77, zero regressions.
bug-1817.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
A live 3-node HA cluster went into a repeated PreVote-failover storm
(term climbing, RDHUP loops, "scheduler not connected") even though
system load was low. Root cause: a hand-typed cleanup command had a
transposed-word typo (xce-ha-deploy vs. the real xce-deploy-ha path)
and silently killed nothing, leaving a full controller cluster alive.
A fresh deploy then bound a second, independent cluster to the exact
same ports, and the two clusters' vote traffic genuinely cross-talked
-- producing symptoms that looked like a real HA bug but weren't.
Two structural gaps let that one typo cascade this badly:
1. deploy-ha.sh had no pre-flight check that target ports were free --
it would happily start a second Raft group on ports an existing one
already held, instead of failing cleanly.
2. start.sh/start-all.sh already had a correct "already running" guard
(reads $TOP/run/*.pid, checks kill -0), but cleanup.sh deleted those
pidfiles unconditionally as part of state cleanup, with no check
that the process each one pointed to was actually dead first --
silently erasing the only signal that guard depends on.
Fixes, all live-verified end to end against real controller/worker/
scheduler/apiserver binaries:
- Added a port pre-flight check before any deploy state is created:
refuses with the exact blocking PID/process if a target port is
already listening.
- cleanup.sh now stops every live PID in $TOP/run/*.pid before
removing any pidfile or state.
- Added stop-all.sh (generated alongside start.sh/cleanup.sh): the
canonical, typo-proof way to stop everything a specific deploy
started, since it reads that deploy's own tracked PIDs instead of
matching process names/paths by hand.
- Found and fixed a related race while testing the above: start-all.sh
didn't wait for worker/scheduler/apiserver to write their pidfiles
before returning "done", so cleanup.sh/stop-all.sh run immediately
after could orphan those two specifically -- the same zombie-process
class this whole fix targets, via a narrower window instead of an
operator typo. Added a short pidfile-appearance poll to close it.
SKILL.md updated to prefer a known deploy's own stop-all.sh over a
hand-typed kill command, document cleanup.sh's new stop-then-wipe
behavior, and add two Troubleshooting rows matching this incident
directly.
bug-1816.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_reconcile_host_resource_usage() -- the per-job mem-usage
reconciliation driven by JOB_RUSAGE events -- was missing the same
IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING guard bug-1801 already added to
sched_update_account(), the regular per-round charge/uncharge path.
core_reset_workers() (bug-1751) deliberately freezes a RECONCILING
worker's resInUse each round so a clean rebuild can restore it once the
worker's real state is confirmed; bug-1801 extended that freeze to the
charge path, but this third, independent resInUse-mutating path kept
applying deltas regardless. Combined with core_events.c's JOBS_PUSH
resync-merge RUN branch, which can replace a job's whole account object
(fresh used=0.0 baseline) without uncharging the old one first -- safe
only because the next clean rebuild is supposed to correct it, exactly
the rebuild that stays skipped while RECONCILING -- a rusage tick
landing in that window kept adding a full recharge on top of whatever
was already frozen in the pool, with nothing to ever subtract it back
out until the worker escaped RECONCILING for good or the job finished.
Live-confirmed via a user's own multi-session replay trace: a job's
"mem used" figure never self-corrected until the job finished.
Added the identical guard, mirroring bug-1801 verbatim. RED confirmed
by temporarily disabling it (exactly the new test failed, "Expected 0
Was 5", 76 others green); GREEN confirmed restored (77/77).
bug-1815, follow-up to bug-1814/bug-1801.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
collect_job_rusages() (xcecd_cgroups.c) reads CPUUsageNSec/MemoryCurrent/
MemorySwapCurrent/TasksCurrent via D-Bus and only checked whether the
round trip itself succeeded (r >= 0). systemd's own convention -- not
previously accounted for here -- is to report UINT64_MAX for a counter
that isn't actually being tracked yet, not an error and not zero.
For MemoryCurrent specifically, an unguarded sentinel read produced
(float)UINT64_MAX / (1024*1024) == 17592186044416.0 (2^44) as a single
job's "memory usage", which the scheduler's rusage reconciliation then
summed across every RUN job that hit the same condition -- a live replay
session showed exactly 19 * 2^44 = 334251534843904.0 ("used=304.00 TB")
for 19 affected jobs, matching to the bit.
Added sd_resource_counter_is_available() and gated all four property
reads on it. Applied uniformly (not just MemoryCurrent) since systemd
uses the identical convention for the other three counters. Left the
existing memory_rss MB wire unit alone (bug-1735's established
contract, also relied on by the controller's own display pipeline) --
changing it would need auditing that separate consumer, out of scope
here.
bug-1814.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce-build.sh previously left --jobs unset with no fallback -- a bare/
direct invocation (CI, a manual `bash xce-build.sh`, anything bypassing
the /xce-build skill) got meson/ninja/go's own unthrottled
auto-parallelism (~nproc+2), risking starvation on this sandbox's
routinely-concurrent build/test sessions.
The script now computes the same live default the /xce-build skill has
always computed by hand -- nproc, available memory, and load average
read fresh at invocation, with a per-profile memory budget (1536 MB/job
asan, 1024 MB/job debug/release) and process-factor discount (0.5 asan/
debug, 0.7 release) for the compiler driver's forked cc1/as children --
whenever --jobs isn't passed explicitly. An explicit --jobs N still
always wins; skill-driven builds are unaffected since the skill already
passes its own computed value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.
The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.
test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
A resync redelivery (on_scheduler_join()'s full push, fired on every
scheduler reconnect) can race a job's own very-recent dispatch
confirmation and carry a stale, non-terminal state for it. The
duplicate-jobid merge in core_events.c trusted this claim
unconditionally, freeing the job's already-captured allocation and
regressing its state -- live-confirmed via a production scheduler.log
showing a resync WARN landing 16ms after the job's own RUN
confirmation, and matching a live-reported simulator symptom (a RUN
job silently missing from info worker's allocation listing).
Refuse a non-terminal claim when the existing job already has a real
allocation on record, mirroring the existing phantom-RUN guard's own
reasoning in the opposite direction; a genuinely terminal claim
(DONE/EXIT) still applies and releases the allocation correctly, same
as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
nproc - load_avg treats one ninja job as one process, but the compiler
driver forks cc1/cc1plus then as, and asan/debug keep those children
resident far longer than release's leaner ones -- under-throttling by
roughly that factor. Apply a per-profile PROCESS_FACTOR (0.5 for
asan/debug, 0.7 for release) to the CPU term before taking the min with
the memory-budget term.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
pthread_cond_wait() reacquires its mutex before honoring a cancellation,
but mylock.c installed no cleanup handler to release it again -- a
thread cancelled mid-wait terminated holding the lock forever. vtaskmgr's
pthread_cancel()-based teardown could land there from three angles
(dispatcher cancelled while polling the shared queue, or a vcpu racing
back into it after its own shutdown signal), permanently deadlocking any
other thread that touched the same queue and hanging vtaskmgr_fini()'s
pthread_join() indefinitely.
Reproduced live as xce_test hanging on Ctrl-D exit once a connection
context was entered; gdb confirmed a vcpu thread blocked forever on a
mutex owned by an already-exited thread ID.
Fix: cbuf_put()/cbuf_get()/cbuf_timedget() now unlock via a
pthread_cleanup_push/pop handler around the wait, so a cancelled waiter
always releases the mutex. vcpu_idle()/vcpu_entry() also recognize the
teardown poison pill and exit immediately instead of needlessly
re-announcing into the already-dead shared queue.
New test_vtaskmgr_teardown.c reproduces the race deterministically
(self-pinned to one CPU, since true multi-core parallelism rarely hits
the window) and bounds vtaskmgr_fini() with a timed join so a
regression fails fast instead of hanging the suite. Logged as bug-1810.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce_core_push_events() WARNed on every JOBS_PUSH resync redelivery
(sched_add_job rc=-EEXIST), which is the expected outcome under
on_scheduler_join()'s full resync or the array-push dedup race
(bug-1734/bug-1714) and flooded operational logs. Only that case now
logs via a new DBG() macro (utils.h); any other add_rc (e.g. -ENOMEM)
still WARNs. DBG is named to avoid colliding with the build's own
-DDEBUG command-line define (debug/asan buildtypes).
Also collapsed hash_string()'s duplicated djb2 loop to delegate to
hash_block(), which already implements the same RUN_HASH loop.
Logged as bug-1809 in .wolf/buglog.json.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
