repl_info_limit() now prints an explanatory note when a limit is implicitly
host-scoped (is_host_based_limit() > 0 due to a percent-based value on a
non-shared resource) but its own configuration doesn't show an explicit
per-host scope. The real discriminator turned out to be consumers[CONSUMER_
HOST]->per_consumer, not pointer-nullity as design.md's gloss said --
build_consumer() unconditionally allocates a consumer for all four
dimensions, so the struct field is never actually NULL in production. Fixed
using the same field is_host_based_limit() itself already falls back to;
verified via RED-phase reproduction that the literal design.md condition
never fires. design.md corrected to document the real condition.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now prints a matched limit's configured setting on both the
all and named-lookup success paths: fmt_limit() (existing libxce formatter,
unmodified) into a stack buffer, re-split and printed through a new
repl_print_limit_dump() helper matching this file's existing [sim]-prefix-
per-line convention. Inclusion and exclusion group membership both render
since fmt_group() already handles both, adopted as-is per design.md's
explicit reuse decision.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() (repl_info.c, not yet wired into repl_cmd_info() -- that's
task 4) resolves "all" vs a specific name against limit_get_configured_limits():
a NULL/empty handle on the all path reports "no limits configured" (Req 1.7);
a NULL handle or key_find() miss on the named path reports "no such limit
'<name>'" (Req 1.3) and never collapses into the all-only message. New
white-box test_repl_limit_table.c exercises both paths directly, including a
dedicated regression guard for the NULL-scoping rule.
Making repl_info.c genuinely call limit_get_configured_limits() broke the link
for ~24 other white-box test files that #include repl_info.c without also
linking plugins/limit.c -- fixed with the same link-satisfying-stub idiom this
file family already uses elsewhere (all return NULL, verified behavior-
preserving since none of these fixtures register the limit plugin).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
limit_get_configured_limits()/limit_get_usage_instances() cross the limit
plugin's private-state boundary (plugin->data, previously unreachable outside
limit.c) via key_find on the name-keyed plugin registry, returning NULL when
the plugin is absent/uninitialized and a raw iterable kvmgr handle otherwise.
Pure reads only -- no allocation, mutation, or hook triggering. Foundation for
the upcoming `info limit` REPL command (task 1 of 5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1820's own fix assumed sudo always forks exactly one direct child
that IS the real target, and looked for it via `pgrep -P $PID` (direct
children only). On a real host, sudo commonly interposes an additional
"monitor" process between the invoked sudo and the actual target
(sudo -> monitor -> target, three processes, not two -- standard
sudo 1.8.7+ architecture for signal/exit-status relay) -- a shape this
sandbox's own simpler sudo build doesn't reproduce, which is why
bug-1820 verified clean here but still left the real worker orphaned
on the reporting user's box. `pgrep -P` found and killed the monitor
(a direct child) while the actual worker (the monitor's own child, not
a direct child of the pidfile'd PID) survived untouched.
Replaces the parent/child tree-walking approach entirely with a
depth-independent identity-based re-scan: both generated scripts now
search system-wide (pgrep -f) for any still-running process whose own
cmdline matches the worker's distinguishing identity (-n $NAME plus
this deploy's -l $TOP/logs -- the same two args start-worker.sh's own
"already running" guard checks) and kill every match directly, however
many hops deep it landed. No longer needs to know or guess sudo's
internal process-tree depth.
Live-verified: regression-checked against this sandbox's own real
(2-level) sudo deploy (still clean), and built + killed a synthetic
3-level process tree modeled on the user's exact live report,
confirming the deepest process is found directly and the intermediate
hop then exits on its own once its child is gone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xceadmin's --controller option (config push, and scheduler record
start/stop/status) accepted only a single host:port. Passing every
configured controller as a comma-separated list -- the natural way to
query an HA cluster without knowing which node is alive/primary --
mis-parsed the whole joined string as one bogus hostname and failed
with a DNS-resolution error.
Adds xceadmin.discovery.resolve_controllers() (comma-split candidate
parsing) and follow_redirects_any() (tries each candidate in order,
skipping ones that are unreachable at the network level -- connection
refused, DNS failure, or a discovery timeout -- mirroring xcecli/
xce_test's own client-side master-rotation failover). A candidate that
IS reachable but returns a genuine protocol-level failure is not
retried against a different node. push.py and scheduler.py's own
duplicated single-host _split_controller helpers are removed in favor
of this shared implementation.
Also fixes a related gap bug-1821's own live verification surfaced:
right after a primary is killed, a standby doesn't start a new
election until its own heartbeat timeout elapses (~10s), so it keeps
redirecting every candidate to the now-dead old primary -- exhausting
the whole candidate list and raising NoPrimaryError, which
_send_admin_request()'s existing transient-retry loop (bug-1713/1715)
never caught. That loop now retries on NoPrimaryError too, using its
own (longer) budget sized for the heartbeat-timeout-plus-election
window rather than the RPC-level "already electing" budget the
original retry logic was tuned for.
TDD RED->GREEN, 304/304 tests passing. Live-verified end to end: a
comma-list correctly skips a dead first candidate and reaches a live
one on a healthy cluster, and correctly rides out a real primary kill
and re-election (retry loop visibly cycling ~15s) to reach the newly-
elected primary rather than failing immediately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
start-worker.sh's real-worker branch writes the pidfile via `echo $$`
then `exec sudo "$WORKER_BIN" ...` -- exec replaces the pidfile-writing
shell with sudo in place (same PID), so the pidfile records sudo's own
PID, not the actual xcecd daemon's. sudo forks a separate, root-owned
child to run the real worker; kill -9 on sudo's PID kills only sudo
(SIGKILL can't be forwarded to a child), silently orphaning xcecd. The
existing "still alive after kill -9" warning never caught this either,
since it re-checked the same (already-dead) sudo PID.
Both generated stop-all.sh and cleanup.sh now detect a sudo-wrapped
pidfile entry (its /proc/$PID/cmdline starts with "sudo "), find its
child via pgrep -P, and kill that child with sudo first. The
still-alive check now also tests for the child's continued existence.
Live-verified end-to-end: deployed a 3-node cluster with a real
(XCE_REAL_WORKER=1) worker, confirmed the sudo-parent/root-child PID
pair, ran the fixed stop-all.sh and cleanup.sh and confirmed zero
processes remain after each, with the new "is a sudo wrapper" line
printed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Found live during task 8.5's optional catching-up-observation attempt:
a killed-then-reconnected standby can end up occupying a NEW
replicator_t.standbys[] slot while its OLD slot (host/port still
populated, ep == NULL) also matches the same configured peer.
ctrlstatus_fill's primary-side peer-status loop stopped at the first
host:port match, so it permanently reported that peer dead even
minutes after it had genuinely reconnected and fully caught up --
violating Requirement 13.3's "true, currently-known connection status"
guarantee.
Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot
with ep != NULL over a slot with ep == NULL; fall back to dead only
when no matching slot is live. Read-only, lock scope unchanged from
task 8.2.
TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced
by the reviewer (exact RED replay) plus two independent live
kill/backlog/restart/poll cycles on separate fresh clusters, both
confirming the previously-stuck standby now converges to ok.
design.md's Locked Decision 3 corrected to document the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds ctrlstatus_peer_status_str (mirrors ctrlstatus_role_str's
convention: 0=unknown/1=ok/2=catching-up/3=dead, safe default fallback
for any other code) and extends cmd_ctrlstatus's per-controller print
line to render it alongside each configured controller's existing
name:address, completing the CtrlPeer.status wire field added in task
8.1 and sourced in task 8.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Populates CtrlPeer.status (added in task 8.1) inside ctrlstatus_fill's
peer-building loop: self always reports ok; a primary responder sources
every other peer from the live replication connection registry
(replicator_t.standbys[], host+port matched, reusing
repl_active_standby_count's ep!=NULL && !in_catch_up predicate) --
dead/catching-up/ok; a non-primary responder reports unknown for every
non-self peer. A self-identity mismatch (pre-existing misconfiguration)
logs one WARN and falls through to the normal non-self sourcing rather
than forcing any peer to ok. Read-only: a bounded lock-protected scan,
no state mutation.
TDD: 3 new tests (primary-sourced-from-registry, standby-unknown-except-
self, self-mismatch-falls-through) plus an extended no-mutation
assertion; RED->GREEN independently reproduced by the reviewer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlPeer.status (UInt8, next free ordinal @3 after port) plus the
matching xce_ctrlpeer_t.status typemap field, verified field-for-field
against the regenerated codec with a genuine encode/decode round-trip
test. No sourcing/rendering logic yet (task 8.2/8.3) -- field defaults
to 0 (unknown) until wired.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Live 3-node E2E validation (task 7.6) found that cmd_ctrlstatus's
electedTerm/electedAt lines (task 7.5) gated on NULLSTR(primaryName),
which is unconditionally empty on every non-primary responder in this
build (ctrlstatus_fill only self-reports the primary's own name --
ha->primary_host is vestigial, a pre-existing Req 2 behavior). This made
electedTerm/electedAt print "unknown" on every standby, forever, even
though elected_at_ms was correctly, non-zero, and convergently populated
the whole time (term/committedLsn/appliedLsn, sourced the same
unconditional way, DID correctly display and converge on the same query).
Fix: gate on status->elected_at_ms == 0 instead -- the field stays
exactly 0 until a real promotion stamps it (either road) or a heartbeat
carries a learned value from an actual primary, unlike term (which can
legitimately bump from mere vote/candidacy activity before any primary
is elected). ctrlstatus_fill itself is untouched; this is purely a
cmd_ctrlstatus display-layer correction.
Also corrects design.md's "Design Delta -- Requirement 12" Locked
Decision 3 (and its dependent traceability/service-contract text) to
document this live-E2E finding, mirroring the spec's existing delta
convention (Req 10/11's own live-E2E correction).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
cmd_ctrlstatus (src/harness/test.c) now prints electedTerm/electedAt
right after the Primary: line, gated on the same NULLSTR(primaryName)
"unknown" check that line already uses. electedTerm prints the existing
term field verbatim (no new wire field, per design's Locked Decision 1);
electedAt reuses the existing ctrlstatus_format_time_ms formatter
already used for responderTime, falling back to "unknown" on a NULL
format result. No leakage of a stale term/elected_at_ms when no primary
is known.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
ctrlstatus_fill now unconditionally reads ha->last_known_elected_at_ms
into the reply's elected_at_ms field -- one atomic-acquire load, no
role-branch, mirroring how the handler already reports raw term
unconditionally. Serves both a self-stamped primary and a
heartbeat-learned standby from the same field. "Unknown" display
gating is deliberately left to the client command (task 7.5), not
this handler.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The primary echoes its own last_known_elected_at_ms on every outgoing
heartbeat (threaded through xce_ha_heartbeat_req's existing scalar-arg
plumbing, alongside term/committed_lsn/committed_config_tag); a standby
processing a received heartbeat (ha_consume_heartbeat_committed, the sole
production call site) stores it unconditionally into its own
last_known_elected_at_ms, reconcile-free -- no other HA field moves.
ha_observe_heartbeat_committed itself is untouched; the new store sits
next to its call site, not inside it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Stamps ha_state_t.last_known_elected_at_ms (CLOCK_REALTIME, epoch-ms) in
both ha_do_promote (non-raft road) and raft_cb_become_leader (raft road),
placed at each road's genuine, final commit point to CTRL_PRIMARY -- past
every early-return refusal path in raft_cb_become_leader (first-boot
phase2 failure, the Live Apply Resource Fence re-check, the promotion
eligibility gate, config-serve-fenced, upgrade-materialization-fenced).
Purely observational: does not change whether/when a promotion succeeds
or is refused. Three new tests (2 positive, 1 negative) prove placement:
a refused promotion leaves the field at its pre-attempt value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlStatus.electedAtMs@14 and HeartbeatMsg.electedAtMs@4 to the
capnp schema, their hand-authored typemap.h counterparts, and a new
ha_state_t.last_known_elected_at_ms field alongside last_known_committed_lsn.
Schema/state plumbing only -- nothing stamps or consumes these fields yet
(tasks 7.2-7.5). Codec round-trip tests confirm both new fields survive
encode/decode unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
A user-reported core dump showed xce_scheduler crashing with SIGSEGV in
its allocator thread: zarray_get_ex dereferencing a NULL array->data
inside match_job_host_limit_() (plugins/limit.c), reached via schedobj.c's
reset_sched_obj().
Root cause: zarray_fini() freed array->data and set it to NULL, but left
array->caps/count/pos at their stale pre-fini values instead of resetting
to the safe (0,0,0) state zarray_init() establishes. xlist_fini() had the
identical gap. Both structs' own bounds checks (`idx >= caps`) treat a
nonzero caps as a guarantee that data is a valid allocation -- a
guarantee broken for any caller that reuses the same struct after
_fini() without re-init.
schedobj.c's reset_sched_obj() does exactly that: it tears down a job's
own PERSISTENT schedobj (reused for the job's whole lifetime, not a
disposable scratch object) whenever the cluster's config version has
advanced, with no matching re-init. Once a host-limit match had grown
hlimits' capacity, and the config version bumped again while that job
sat PEND, the next allocation attempt for the same job dereferenced the
torn array -- exactly matching the reported backtrace. This same crash
class had already caused at least one earlier, never-successfully-root-
caused crash near the same call site (2026-08-17).
Fixed at the library-primitive level rather than reset_sched_obj()'s
call site: zarray_fini()/xlist_fini() now reset caps/count/pos to 0,
matching what _init() establishes. This makes every existing
_fini()-then-reuse caller correct by construction, with zero changes
needed at any call site. Audited every non-test zarray_fini()/
xlist_fini() call site first -- none relies on caps/count surviving
_fini(), so this is a pure safety improvement, not a behavior change.
RED confirmed by temporarily disabling both resets: new tests failed
cleanly ("Expected 0 Was 4" / "Expected 0 Was 2", the stale pre-fini
caps). GREEN confirmed restored: full lib suite 314/314, scheduler
suite 77/77, zero regressions.
bug-1817.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
A live 3-node HA cluster went into a repeated PreVote-failover storm
(term climbing, RDHUP loops, "scheduler not connected") even though
system load was low. Root cause: a hand-typed cleanup command had a
transposed-word typo (xce-ha-deploy vs. the real xce-deploy-ha path)
and silently killed nothing, leaving a full controller cluster alive.
A fresh deploy then bound a second, independent cluster to the exact
same ports, and the two clusters' vote traffic genuinely cross-talked
-- producing symptoms that looked like a real HA bug but weren't.
Two structural gaps let that one typo cascade this badly:
1. deploy-ha.sh had no pre-flight check that target ports were free --
it would happily start a second Raft group on ports an existing one
already held, instead of failing cleanly.
2. start.sh/start-all.sh already had a correct "already running" guard
(reads $TOP/run/*.pid, checks kill -0), but cleanup.sh deleted those
pidfiles unconditionally as part of state cleanup, with no check
that the process each one pointed to was actually dead first --
silently erasing the only signal that guard depends on.
Fixes, all live-verified end to end against real controller/worker/
scheduler/apiserver binaries:
- Added a port pre-flight check before any deploy state is created:
refuses with the exact blocking PID/process if a target port is
already listening.
- cleanup.sh now stops every live PID in $TOP/run/*.pid before
removing any pidfile or state.
- Added stop-all.sh (generated alongside start.sh/cleanup.sh): the
canonical, typo-proof way to stop everything a specific deploy
started, since it reads that deploy's own tracked PIDs instead of
matching process names/paths by hand.
- Found and fixed a related race while testing the above: start-all.sh
didn't wait for worker/scheduler/apiserver to write their pidfiles
before returning "done", so cleanup.sh/stop-all.sh run immediately
after could orphan those two specifically -- the same zombie-process
class this whole fix targets, via a narrower window instead of an
operator typo. Added a short pidfile-appearance poll to close it.
SKILL.md updated to prefer a known deploy's own stop-all.sh over a
hand-typed kill command, document cleanup.sh's new stop-then-wipe
behavior, and add two Troubleshooting rows matching this incident
directly.
bug-1816.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_reconcile_host_resource_usage() -- the per-job mem-usage
reconciliation driven by JOB_RUSAGE events -- was missing the same
IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING guard bug-1801 already added to
sched_update_account(), the regular per-round charge/uncharge path.
core_reset_workers() (bug-1751) deliberately freezes a RECONCILING
worker's resInUse each round so a clean rebuild can restore it once the
worker's real state is confirmed; bug-1801 extended that freeze to the
charge path, but this third, independent resInUse-mutating path kept
applying deltas regardless. Combined with core_events.c's JOBS_PUSH
resync-merge RUN branch, which can replace a job's whole account object
(fresh used=0.0 baseline) without uncharging the old one first -- safe
only because the next clean rebuild is supposed to correct it, exactly
the rebuild that stays skipped while RECONCILING -- a rusage tick
landing in that window kept adding a full recharge on top of whatever
was already frozen in the pool, with nothing to ever subtract it back
out until the worker escaped RECONCILING for good or the job finished.
Live-confirmed via a user's own multi-session replay trace: a job's
"mem used" figure never self-corrected until the job finished.
Added the identical guard, mirroring bug-1801 verbatim. RED confirmed
by temporarily disabling it (exactly the new test failed, "Expected 0
Was 5", 76 others green); GREEN confirmed restored (77/77).
bug-1815, follow-up to bug-1814/bug-1801.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
collect_job_rusages() (xcecd_cgroups.c) reads CPUUsageNSec/MemoryCurrent/
MemorySwapCurrent/TasksCurrent via D-Bus and only checked whether the
round trip itself succeeded (r >= 0). systemd's own convention -- not
previously accounted for here -- is to report UINT64_MAX for a counter
that isn't actually being tracked yet, not an error and not zero.
For MemoryCurrent specifically, an unguarded sentinel read produced
(float)UINT64_MAX / (1024*1024) == 17592186044416.0 (2^44) as a single
job's "memory usage", which the scheduler's rusage reconciliation then
summed across every RUN job that hit the same condition -- a live replay
session showed exactly 19 * 2^44 = 334251534843904.0 ("used=304.00 TB")
for 19 affected jobs, matching to the bit.
Added sd_resource_counter_is_available() and gated all four property
reads on it. Applied uniformly (not just MemoryCurrent) since systemd
uses the identical convention for the other three counters. Left the
existing memory_rss MB wire unit alone (bug-1735's established
contract, also relied on by the controller's own display pipeline) --
changing it would need auditing that separate consumer, out of scope
here.
bug-1814.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce-build.sh previously left --jobs unset with no fallback -- a bare/
direct invocation (CI, a manual `bash xce-build.sh`, anything bypassing
the /xce-build skill) got meson/ninja/go's own unthrottled
auto-parallelism (~nproc+2), risking starvation on this sandbox's
routinely-concurrent build/test sessions.
The script now computes the same live default the /xce-build skill has
always computed by hand -- nproc, available memory, and load average
read fresh at invocation, with a per-profile memory budget (1536 MB/job
asan, 1024 MB/job debug/release) and process-factor discount (0.5 asan/
debug, 0.7 release) for the compiler driver's forked cc1/as children --
whenever --jobs isn't passed explicitly. An explicit --jobs N still
always wins; skill-driven builds are unaffected since the skill already
passes its own computed value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.
The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.
test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
A resync redelivery (on_scheduler_join()'s full push, fired on every
scheduler reconnect) can race a job's own very-recent dispatch
confirmation and carry a stale, non-terminal state for it. The
duplicate-jobid merge in core_events.c trusted this claim
unconditionally, freeing the job's already-captured allocation and
regressing its state -- live-confirmed via a production scheduler.log
showing a resync WARN landing 16ms after the job's own RUN
confirmation, and matching a live-reported simulator symptom (a RUN
job silently missing from info worker's allocation listing).
Refuse a non-terminal claim when the existing job already has a real
allocation on record, mirroring the existing phantom-RUN guard's own
reasoning in the opposite direction; a genuinely terminal claim
(DONE/EXIT) still applies and releases the allocation correctly, same
as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
pthread_cond_wait() reacquires its mutex before honoring a cancellation,
but mylock.c installed no cleanup handler to release it again -- a
thread cancelled mid-wait terminated holding the lock forever. vtaskmgr's
pthread_cancel()-based teardown could land there from three angles
(dispatcher cancelled while polling the shared queue, or a vcpu racing
back into it after its own shutdown signal), permanently deadlocking any
other thread that touched the same queue and hanging vtaskmgr_fini()'s
pthread_join() indefinitely.
Reproduced live as xce_test hanging on Ctrl-D exit once a connection
context was entered; gdb confirmed a vcpu thread blocked forever on a
mutex owned by an already-exited thread ID.
Fix: cbuf_put()/cbuf_get()/cbuf_timedget() now unlock via a
pthread_cleanup_push/pop handler around the wait, so a cancelled waiter
always releases the mutex. vcpu_idle()/vcpu_entry() also recognize the
teardown poison pill and exit immediately instead of needlessly
re-announcing into the already-dead shared queue.
New test_vtaskmgr_teardown.c reproduces the race deterministically
(self-pinned to one CPU, since true multi-core parallelism rarely hits
the window) and bounds vtaskmgr_fini() with a timed join so a
regression fails fast instead of hanging the suite. Logged as bug-1810.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce_core_push_events() WARNed on every JOBS_PUSH resync redelivery
(sched_add_job rc=-EEXIST), which is the expected outcome under
on_scheduler_join()'s full resync or the array-push dedup race
(bug-1734/bug-1714) and flooded operational logs. Only that case now
logs via a new DBG() macro (utils.h); any other add_rc (e.g. -ENOMEM)
still WARNs. DBG is named to avoid colliding with the build's own
-DDEBUG command-line define (debug/asan buildtypes).
Also collapsed hash_string()'s duplicated djb2 loop to delegate to
hash_block(), which already implements the same RUN_HASH loop.
Logged as bug-1809 in .wolf/buglog.json.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Every cmd_* REPL handler in test.c/join_worker_args.c reset flag parsing
with `optind = 1;` only. glibc's getopt_long only fully reinitializes its
internal scan state (nextchar/first_nonopt/last_nonopt) when optind==0, or
on the process's very first getopt_long() call ever -- optind=1 alone
leaves that state stale for every call after the first. Since xce_test's
REPL dispatches many independent cmd_*() calls per process, a command
whose parse ends on a short no-argument flag (`qctrl -a`, `hostctrl -o`)
left state that corrupted the NEXT unrelated command's parse badly enough
to misread its own name as an unrecognized flag. This was the actual
cause of harness_unit_test's 4 deterministic (not flaky -- 20/20 identical
repeat runs) test_test_cli.c failures, each burning its full 5s
confirmation-prompt timeout. Fixed by using optind=0 at all 21 reset
sites; confirmed via 3 isolated pty repros plus 10 consecutive clean
harness_unit_test runs (132/132, was 4 failures/run every time).
Also fixes a resource leak found while investigating: test_test_cli.c's
spawned xce_test sessions were only torn down by inline
clitest_kill_and_reap()/clitest_close() calls at the bottom of each test
function, which Unity's longjmp-based TEST_ASSERT abort skips entirely on
failure, leaking the child process and its pty/pipe fds. Added a static
session mirror (safe across the longjmp, unlike a stack pointer) reaped
from harness_test.c's tearDown(), which Unity guarantees runs after every
test regardless of outcome.
Logged as bug-1807/bug-1808 in .wolf/buglog.json, with the underlying
getopt/longjmp gotchas recorded in .wolf/cerebrum.md. Full ASAN
/xce-build harness also passes 5/5 with no sanitizer findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Follow-up to the xce-harness-component spec, which relocated xce_test
from src/controller/testsuite/ into a standalone src/harness/
component (commits cf55d691..cf380750). These four files were outside
that spec's own file boundary (owned by other specs / general tooling)
so were deliberately left alone at the time, but their hardcoded
default path and error-message advice were now stale and would have
misled anyone who hit them:
- .claude/skills/xce-build/SKILL.md: added `harness` to both component
lists (frontmatter description + args section), and replaced the
"controller always builds xce_test too" claim with an accurate
description of the harness component, its own build command, and
the accepted Meson test-runner quirk (bug-1806) that can still
transitively rebuild xce_test as a side effect of another
component's default-profile test run.
- src/controller/testsuite/test_rcfile_integration.sh (owned by the
xce-test-rc-file spec): XCE_TEST_BIN default and failure message.
- src/controller/testsuite/clean_failover_preservation.sh (owned by
ha-term-safety task 5.2): same.
- src/tools/test-commit-durability.sh: same, plus its header comment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Feature-level validation (/kiro-validate-impl) found src/harness/CLAUDE.md
(written by task 1.3, before tasks 2.1/2.2/3.1 landed) still claimed no
/xce-build harness entry point and no unit-test binaries existed -- both
false by the time all 16 tasks completed. Updated to describe the real,
final state: /xce-build harness as the build command, and all 5 test
binaries (harness_unit_test + 4 standalone) under the harness suite.
Also logs one more out-of-boundary stale-path finding
(src/tools/test-commit-durability.sh:97) for a future owner.
This closes out feature-level validation for xce-harness-component --
mechanical (173+1213 tests, 0 failures), design/boundary alignment
(zero controller-private dependency, both sanctioned exceptions exact
match), and requirements coverage (22/22 mapped) all passed cleanly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Add a harness) case to src/tools/xce-build.sh following the same
three-part template (setup_build / meson compile / conditional meson
test) already used by scheduler)/worker), building all 6 harness
targets (xce_test plus its 5 test binaries) and running suite harness.
Remove controller/xce_test from the controller) case's compile target
list -- it no longer exists as a controller target. Add harness to the
script's usage/component-list text.
Verified both directions: `/xce-build harness` alone produces xce_test
and all 5 test binaries (4/20/5/12/132 tests, 0 failures); `/xce-build
controller` alone no longer produces controller/xce_test while
controller_unit_test stays green at 1199 tests.
Task 3.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Move test_resval_typeresolve.c, test_joblist_display.c,
test_join_worker_args.c, and test_listrow_json.c into
src/harness/testsuite/, and re-create their executable()/test() blocks
verbatim in src/harness/meson.build under suite 'harness' (task 1.1 had
already fully deleted them from src/controller/meson.build, not just
moved them, so these had been orphaned on disk since that commit).
Fix test_joblist_display.c's ctrlstatus_fmt.h include path to the
xce/ location promoted in task 1.2.
Verified against the pre-task-1.1 baseline (git show cf55d691^): all
four blocks' source lists, c_args, and env are faithfully reproduced,
only paths and the suite tag changed.
harness suite: 5 binaries, all green (resval_typeresolve 4, joblist_display
20, join_worker_args 5, listrow_json 12, harness_unit_test 132 -- 173 total,
0 failures). controller suite unchanged: 1199 Tests 0 Failures 0 Ignored.
This completes Major Task 2 (Core) -- the harness component now owns all
5 of its test binaries independently of the controller build.
Task 2.2 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Relocate the 13 test files that lived inside controller_unit_test into
src/harness/testsuite/, add a new Unity runner (harness_test.c, modeled
on controller_test.c) carrying their extern/RUN_TEST entries, and
register the new harness_unit_test executable + suite in
src/harness/meson.build with an explicit build-order dependency on
xce_test (two of its tests spawn that binary over a pty).
Finish clearing the controller side: remove test_test_cli.c/
test_xcetest_process_entry.c from controller_unit_test_src and their
extern/RUN_TEST entries from controller_test.c, and remove the
temporary xce_test_controller_link symlink bridge task 1.1 added (no
longer needed now that both binaries build as siblings under harness/).
Verified: harness suite 132 Tests 0 Failures 0 Ignored; controller suite
1199 Tests 0 Failures 0 Ignored. 1199+132=1331 matches the pre-move
controller_unit_test baseline exactly (cross-checked against buglog
bug-1798/bug-1805).
Task 2.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
New src/harness/CLAUDE.md documenting the component's purpose, current
build command, and file layout -- every claim grounded in the live repo
state (verified: no /xce-build harness entry point yet, no
harness_unit_test binary yet, both land in later tasks of this spec).
Task 1.3 of xce-harness-component -- completes the Foundation phase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Move the one genuinely shared header (header-only, static inline, zero
link dependency) from src/controller/testsuite/ to src/include/xce/,
alongside the existing cross-daemon shared headers (cluster.h, types.h,
errnox.h). Repoint its three current consumers (harness's test.c and
joblist_display.c, controller's test_ctrlstatus.c) to the new path.
test_joblist_display.c is intentionally left on the old path -- it
relocates in task 2.2.
Also removes src/harness/meson.build's temporary '../controller/testsuite'
incdir shim, which task 1.1 added and explicitly marked for removal once
this task landed.
controller_unit_test: 1228 Tests 0 Failures 0 Ignored (unchanged from
task 1.1's baseline, confirming a header-path-only change).
Task 1.2 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Move xce_test's 15 production sources + headers from src/controller/testsuite/
into a new src/harness/ component with its own meson.build, and strip every
now-dangling controller-side reference (11 controller_unit_test_src entries,
4 standalone test-binary blocks, 11 controller_test.c extern/RUN_TEST lines)
so the controller build is never left broken.
Fixes a broken relative-symlink bridge (xce_test_controller_link) added to
keep two controller-embedded tests (test_test_cli.c, test_xcetest_process_entry.c)
finding the relocated xce_test binary until task 2.1 moves them too.
Discovered and documented in design.md: four controller-domain regression
tests call functions directly from four of xce_test's own files, requiring
controller_unit_test to permanently recompile them from src/harness/ — a
real dependency neither the original discovery nor design audit caught.
controller_unit_test: 1228 Tests 0 Failures 0 Ignored (verified via the
Unity binary's own printed summary line, not just meson's aggregate count).
Task 1.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's
per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way
core_reset_workers already does (bug-1751), closing the gap that let a
worker's used slots exceed its own capacity across a resync replay
- bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so
a captured job's job->state is set to RUN synchronously at dispatch capture,
mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead
of deferring until its separately-recorded JOB_STATE_CHANGE event replays
- bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the
same human-scaled unit formatting already used for maxmem/maxswap/maxtmp
- bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after
live feedback that it over-fires inside busy sessions; replace it with a
prescan-based "next non-dummy session" hint in info summary instead
Full scheduler suite: 77/77 tests pass (asan profile).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The hand-written per-flag description explaining -n's <min>[,<max>]
format was dropped when task 3.5 moved the flag list to the generated
flagspec block (which only shows short/long/required/default, with no
room for a flag's value-format semantics). Restore the explanation as
prose in USAGE_JOBSUB_HEAD, following the same pattern already used by
resupdate's HEAD (-n/-v pairing, -t's conditional meaning).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Follow-up audit (requested: check other generated sources, specifically
packcc/PEG output) found the identical bug shape in the four PEG-parser
custom_targets: section_parser, rusage_parser, xce_resreq_parser, and
xce_select_parser each produce a .c + a paired .h (section.h, rusage.h,
xce_resreq.h, xce_select_peg.h), but libxce_src only referenced each
parser's [0] (.c) index. lib/resreq.c -- confirmed via a project-wide
grep to be the only consumer of any of these four headers -- had no
declared build-order dependency on any of the four codegen steps.
Fix: pass each of the four custom_targets unindexed into libxce_src,
same pattern as the capnp fix. No libxce_dep change needed here since
nothing outside src/lib/ consumes these headers.
A project-wide search confirms these, plus the capnp custom_target, are
the only multi-output C/H codegen steps in the whole build -- every other
custom_target() (xceadmin, apiserver, xcecli, autoscaler/providers)
produces a single Go-binary/stamp output with no equivalent hazard.
Verified: a third consecutive full from-scratch `./tools/xce-build.sh all
debug` build (1173/1173 targets) succeeded clean; a fresh `lib debug`
build plus the full lib test suite (313/313) passed; a fresh incremental
controller/ASAN rebuild passed 1331/1331 on a quiet re-run (its first,
concurrent-with-the-full-build run hit an unrelated, confirmed-flaky PTY
harness timing failure in test_smoke_launch_send_echo_capture_stdout_and_exit_code,
reproduced as system-load-induced, not caused by this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
The previous fix (654c6e8c) only closed the missing-dependency-edge race
inside the libxce target itself. A live user report on a second
from-scratch `./tools/xce-build.sh all debug` run showed the same race
still hitting scheduler/src/sim_record.c and core_events.c, which reach
controller.capnp.h directly via a raw -Ilib include path with no
declared dependency on the codegen step at all.
Root cause: libxce_dep = declare_dependency(link_with: libxce) -- the
dependency object controller/worker/scheduler all consume uniformly --
only orders a consumer's LINK step after libxce is built; it never
propagated a build-order dependency on the capnp custom_target to a
consumer's own COMPILE steps.
Fix: add `sources: [gen_srcs]` to libxce_dep's declare_dependency() call.
This kwarg exists specifically for generated files that must be built
before any consumer of a dependency compiles, and since every component
already lists libxce_dep in its own dependencies, this one change
propagates the missing edge everywhere with no per-component changes.
Verified: two consecutive full from-scratch `./tools/xce-build.sh all
debug` builds (1173/1173 targets, the user's own exact repro command)
both succeeded with no retry; a subsequent incremental controller/ASAN
rebuild stayed clean (1331/1331 tests, 0 ASAN diagnostics).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
libxce_src listed only gen_srcs[0] (controller.capnp.c) as a source of
the static_library('xce', ...) target, never controller.capnp.h -- which
several library files (utils.c and others) reach only via #include.
Ninja had no build edge forcing the capnp custom_target to finish before
those files compiled, so a from-scratch build could schedule them first,
intermittently failing with "controller.capnp.h: No such file or
directory". This is the same symptom recorded across 5+ prior recurrences
in buglog.json since 2026-07-16, previously worked around by retrying the
build rather than root-caused.
Fix: pass the whole gen_srcs custom_target (not gen_srcs[0]) into
libxce_src -- Meson still compiles only its .c output, but now orders the
entire target's build after both outputs are generated. Verified via two
consecutive from-scratch `xce-build.sh lib debug` builds (no retry
needed), the full lib test suite (313/313), and an incremental controller
asan rebuild (1331/1331, 0 ASAN diagnostics) to confirm no regression.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Add one black-box regression test proving process-level help, in-session
help content, and context-scoped tab completion work correctly when
exercised together in one continuous session (context switches +
completion + help lookups), not just individually as each was already
covered by tasks 1.6/2.3/3.5. This closes the final task of the
xce-test-cli-ux implementation plan (23/23).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Audit every printf() call site in test.c not already covered by the
flagspec-rejection or perror() paths; convert 33 genuine error/warning
messages (expect-jobs, expect-job, wait-primary, help's not-found path,
jobrun/jobsub/jobstart/jobfinish/jobfail validation, resupdate, set) from
stdout to stderr, so a command's successful result data and its
diagnostics never share a stream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Wire flagspec_render_help() into cmd_help via new command_t fields
(flagspec/flagspec_count/usage_tail), so every flag-accepting command's
help composes description -> generated flag block -> example/related
notes, with hand-written per-flag lines removed as a duplicate source of
truth. Add missing purpose descriptions, examples, and confirmation-gate
related notes across ~20 commands; fix bug-1711's "matchs" grammar
(Req 11.2); confirm join's corrected maxmem default (bug-1557, Req 12.2)
and the --json flag (Req 5.4) now render correctly end-to-end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Bring workerlist/joblist/reslist onto the shared flagspec_t library with an
identically-spelled --json flag, wired to the task-2.2 JSON list formatter
via a new pure field-mapping module (listrow_json.c/.h). Default table
output is unchanged; JSON rows carry raw numeric values (flags, timestamps,
memory sizes) rather than their humanized display strings, per Requirement
5.1's "same fields" contract.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Gates jobkill and shutdown unconditionally, qctrl only when action is
QUEUE_CLOSE/QUEUE_INACTIVE (not ACTIVE/OPEN), hostctrl only when
HOST_CLOSE (not OPEN), and upgradeconfig unconditionally -- exactly
Requirement 9.5's enumeration, with jstart/jfinish/jfail (worker
context) correctly untouched. Each gains a -y/--yes bypass flag via
flagspec_t; cmd_set gains -y/+y calling confirm_set_skip(). A decline
maps to a plain `return 0` (success-coded, distinct from a negative
failure return) per confirm.h's own documented contract -- this means
a declined destructive command inside a source'd file under `set -e`
won't trip abort-on-failure, a real but narrow interaction only
reachable when a human is actively declining; left for task 4.3 to
test explicitly. 1321/1321 across the full controller suite (+3),
0 regressions.
Task 3.3 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Routes all 26 blocking xcecli_* call sites across ~24 commands through
xcetest_run_interruptible(), wires xcetest_set_active_connection() at
execute_line()'s dispatch point (active_ctx->ctx, read fresh per
command), and fixes main()'s REPL loop to distinguish a real EOF from
an EINTR-interrupted readline() so Ctrl-C at an empty prompt discards
the line instead of exiting the session.
Live-discovered and fixed bug-1794: fill_job_alloc() never initializes
xce_job_alloc_t's nres_shared/allocs_shared fields -- harmless as stack
garbage before this task, genuine heap garbage once moved into a
heap-allocated ctx block (a necessary consequence of the adapter's
heap-only precondition). Fixed by switching all ~26 call-context
allocations from malloc to calloc uniformly. Also fixed an argv-aliasing
hazard: cmd_hello/cmd_jobckpt's -m values aliased optarg, which
execute_line() frees immediately after return including on the
abandoned path -- now strdup'd into the heap block.
Also closed a safety-critical gap this task's own design introduced
(bug-1795): before this task the REPL was fully synchronous, so
leave_ctx() (called by up/quit) could never run while an RPC was
outstanding. An abandoned worker can now persist past command
completion, so leave_ctx() tearing down the connection could
use-after-free it. Fixed via xcetest_wait_connection_idle(), which
blocks on the same per-connection semaphore the adapter already
tracks, then releases it, proving no worker is active before teardown.
Proven with a deterministic white-box RED->GREEN test, since the race
itself is too timing-dependent for a black-box test to reliably force.
1361/1361 across the full controller suite (+2), 0 regressions.
Task 3.2 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.2, 1.3, 1.4, 1.5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Migrates 17 commands (cmd_hello, cmd_jobinfo, cmd_jobrun, cmd_jobsub,
cmd_resval, cmd_jobkill, cmd_jobckpt, cmd_qctrl, cmd_hostctrl,
cmd_runtasks, cmd_setpend, cmd_jobstart, cmd_jobfinish, cmd_jobfail,
cmd_resupdate, cmd_walcompare, join_worker_parse_attrs) from raw
getopt()/hand-rolled parsing to the shared flagspec_t + getopt_long
mechanism, so every one now uniformly rejects an unrecognized or
malformed flag with a usage summary and, where a close match exists,
a suggestion. cmd_walcompare's hand-rolled parser needed a leading '+'
in its optstring to preserve its pre-migration flags-only-before-
positionals behavior. Also fixes bug-1704: cmd_qctrl's uninitialized
action variable, now sentineled and rejected when unset, mirroring
cmd_hostctrl's existing pattern -- this also makes the destructive-
command gate's -i/-c detection (task 3.3) safe to rely on. Every
existing short flag's letter, value format, and default is unchanged
for a valid invocation. 1357/1357 across the full controller suite
(+4), 0 regressions.
Task 3.1 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4, 10.1, 10.2, 10.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
cmd_join_worker's simulated-worker maxmem default was 64 bytes -- a
leftover from when the field was MB-scale, unfixed when maxmem became
bytes-canonical elsewhere. Now 64 * 1024 * 1024. Extracted the
argument-parsing/defaulting logic into join_worker_parse_attrs()
(join_worker_args.c), since it runs entirely before the function's
network call and is a genuine testable seam -- same precedent as
task 2.5's joblist_display.c. Verified live against a real deployed
cluster (workerlist reports MaxMem: 64.00 MB) plus a permanent 5-test
regression suite. maxswap/maxtmp have the identical stale-default bug
(logged as bug-1793) but are deliberately left unfixed here since
USAGE_JOIN_WORKER's help text currently documents their stale values
accurately -- fixing only the code would create a new docs mismatch;
a locked-in sibling-defaults test documents this for the follow-up.
1353/1353 across the full controller suite (+5), 0 regressions.
Task 2.6 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 12.1
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Eight display/text defects in test.c: worker_status_str's "UNKOWN"
typo, cmd_runtasks's "sequentail" typo (display-only, verified inert
against the real launch_type consumer), cmd_reslist's right-justified
columns and unlabeled hex Flags (now matching workerlist/state_raw's
conventions), joblist's raw-epoch timestamps (now via a format_job_time
helper delegating to the same formatter ctrlstatus already uses) and
combined NTasks column split into MinTasks/MaxTasks, job_state_str's
missing raw-state fallback for ambiguous RUN+EXIT (mirroring
print_jobinfo's existing pattern), and getconf's no-op group/config
list stubs (now genuinely implemented from already-fetched data,
matching the three sibling print functions' convention).
worker_status_str/job_state_str/format_job_time are pure functions with
no networking/REPL/global-state dependency, extracted into a new
joblist_display.c/.h module with a permanent 20-test standalone suite --
following this codebase's own resval_typeresolve.c/misc.c precedent for
testing functions that would otherwise be stranded inside untестable
test.c. cmd_reslist/joblist_query/cmd_getconf's table-printing bodies
inline fetch+print and remain untestable without a live cluster.
1348/1348 across the full controller suite (+20), 0 regressions.
Task 2.5 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 11.1, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
cmd_source reuses rcfile_load() unmodified for line execution, adding
an explicit missing-file check (bypassing rcfile_load's silent-missing
policy, which is correct for its one existing startup-rc-file caller
but wrong for an explicit user command) and a realpath()-canonicalized
in-flight path stack that rejects direct or transitive cycles instead
of hanging. Registered in all four command contexts. Translates
rcfile_load's raw positive "aborted" return to -1 so an enclosing
source line correctly sees a transitively-aborted nested source as
failure, not success (preprocess_and_run only treats negative returns
as failure). Proven via real fixture files driven through the black-box
pty harness: a genuine two-file mutual cycle, a missing path, and
ordered execution across a non-cyclic nested source. 1310/1310
controller suite passing (+3).
Task 2.4 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
