Bugs fixed, discovered across a day of live-operator testing against a
real deployed cluster:
- bug-1584: clock anchors never initialized to real wall-clock values,
timestamps showed system uptime instead of Unix epoch.
- bug-1585: heap-use-after-free in the normal-mode driver's periodic
resval update path -- core_events.c's RESVALS_FULL/_DELTA handlers do a
shallow zarray_put() of the wire-decoded xce_val_t*, but driver.c's
release_translated_payload() unconditionally freed the same decoded
array right after, so the next periodic report for the same
worker+resource triggered a UAF read via kill_sched_resval(). New
regression test: test_resval_ownership.c.
- bug-1587/1588/1591: raw-ns/raw-uint64 jobid/timestamp prints across the
REPL/SIM family converted to the same human-readable formatting already
used elsewhere (fmt_jobid(), timestamp formatting).
- bug-1590: repl_info.c's XCE_VAL_NUMERIC checks corrected to
Value_u_fvalue, fixing `info worker`/`info resource` falling back to
0.00 for real numeric resvals.
task 12.5 (Req 5.22 amendment) -- five REPL gaps surfaced by a live
operator session, each with its own bug id:
- bug-1594: `continue`/`step` reaching EOF with no breakpoint hit now
re-enters the REPL via xce_repl_on_replay_complete() instead of exiting
the process; step/continue issued after replay is already complete
re-prompt instead of silently exiting.
- bug-1595: `step`/`continue` now echo each input event (not just
decisions) via repl_print_event_summary(), in real event/decision
order.
- bug-1593: `info resource <name>|all` now reports resource definition
metadata restricted to XCE_RESF_GLOBAL/XCE_RESF_SHARED resources, read
from core->sharedInst, instead of aggregating host-based resources
across every worker.
- bug-1596: repl_split_tokens() strips one matching pair of surrounding
quote characters from each token.
See requirements.md's Req 5 AC 22 "Amendment (2026-08-19)" for the full
rationale on each of the five task 12.5 gaps.
Full scheduler suite green (41/42; the one remaining failure,
test_recordstart_symlink_escape_replies_error, is the pre-existing
unrelated bug-1560).
fmt_jobid() renders the raw idgen()-packed jobid in the same
cluster-apiserver-timestamp-sequence form already used everywhere else
this codebase logs/displays a jobid (scheduler.c/handler.c, worker's
process_manager.c, xcecli, waldump). job.c's own TRACE lines were the one
place still printing the raw 64-bit packed integer.
Every early-exit branch in the worker_list scan either re-tested the same
node forever (bare `continue`) or returned out of the whole function on the
first DONE/EXITED worker, so job array elements past the head of the list
never got their on_job_start report. Capture `next` before touching each
node and always advance to it. Affects xcecd_virtual and real xcecd alike
(same DBUS_MANAGER-gated function).
Also records a roadmap follow-up (xcecd_virtual concurrent-job support,
MXJ > 1) discovered while investigating this bug.
requirements.md, design.md, research.md, and spec.json were never committed
during implementation (only tasks.md was, per the per-task checkpoint
convention). Committing them now that all 15 tasks are implemented,
reviewed, and validated GO by /kiro-validate-impl.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
verdict-logic-selftest.sh sources the real aggregate_verdict()/
build_verdict_record() (via run-resource-verify.sh's BASH_SOURCE guard, no
live deployment triggered) and exercises them with synthetic per-layer
records: all-match/PASS positive control, a single-resource single-layer
MISMATCH within a multi-resource batch (asserting both correct attribution
of the failure AND that the other resources independently still report
PASS), UNREACHABLE distinct from MISMATCH, REJECTED, and a zero-records
degenerate case. 53 assertions total.
Proven non-tautological: breaking aggregate_verdict()'s comparison logic
(two different ways, independently reproduced by review) makes the selftest
genuinely fail; restoring makes it pass again.
This completes the resource-value-verification spec - all 15 tasks (1.1
through 8.2) implemented, reviewed, and committed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Validation-only, no code changes. Confirmed the complete spec end-to-end
against a freshly built controller+scheduler+apiserver: default 9-resource
set + one deliberately-unregistered name -> VERDICT resources=10
rejected=1 result=FAIL (exit 1), attributable specifically to the rejected
name, not a false PASS; positive control (defaults only) -> VERDICT
resources=9 rejected=0 result=PASS (exit 0). All 27 individual layer-checks
(9 resources x 3 layers) MATCH; raw scheduler.log/apiserver.log trace lines
independently verified against tasks 4.1/4.2's documented format; rejection
confirmed non-silent across 4 independent log sources; cleanup and log
preservation confirmed.
Surfaced (but correctly did not fix, out of boundary) bug-1583: the
apiserver stores resource-location updates with zero name validation, even
for names the controller/scheduler both reject. Confirmed this does not
affect this feature's own correctness - accept/reject is determined
entirely by cmd_resupdate()'s controller-side follow-up query, before any
layer is checked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
aggregate_verdict()/build_verdict_record() combine the three per-layer
check results into a single per-resource comparison, self-contained and
invokable with synthetic records (task 8.2). Uses ASCII Unit Separator
(\x1f) as the record delimiter after discovering bash's IFS tab-splitting
silently collapses adjacent empty fields even with IFS set to only a tab
(bug-1581), which corrupted UNREACHABLE-record field alignment.
Emits one machine-parseable VERDICT line (resources/layers_checked/
mismatches/unreachable/rejected/result) plus the terminal PASS/FAIL oracle
line and matching exit code. Cleanup is the same trap-based mechanism task
5.1 registered for the abort path - logs always preserved regardless of
--keep-running.
Round-2 fix: main()'s live-data wiring (build_verdict_records_from_tsv())
derived a layer's status from array-key existence rather than the actual
recorded outcome - since tasks 6.3/6.4 always set the key (even on a
genuine timeout, just with an empty value), every real UNREACHABLE was
silently misclassified as MISMATCH. Fixed to check SCHEDULER_CHECK_RESULTS/
APISERVER_CHECK_RESULTS string content instead (bug-1582). Live-confirmed
by killing the scheduler mid-run: VERDICT now correctly reports
unreachable=9 mismatches=0, not the reverse.
SKILL.md rewritten to describe the now-complete, functional tool.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_apiserver_log()/query_apiserver_resvals() mirror task 6.3's
scheduler-check pattern: bounded-wait polling (batched across all pending
resources per iteration) of apiserver.log for task 4.2's "resource-location
stored" trace line, parsing the trailing zap JSON block. Reports MATCH /
MISMATCH (both values) / UNREACHABLE (timeout, distinct from mismatch),
tracked in APISERVER_OBSERVED/APISERVER_CHECK_RESULTS +
apiserver-check-results.tsv.
This completes all three per-layer checks. Live-confirmed: the full default
resource set (including cpuf, post-bug-1579-fix) MATCHes simultaneously at
the controller, scheduler, and apiserver layers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously worked around (task 6.2) by removing the one host-attrs resource
(cpuf) from the default set - a custom resource=value arg using any
host-attrs name would still false-mismatch at the controller layer, since
inject_resources() disconnected the worker before the separate controller-
check session ran, and the controller resets host-attrs resources on
disconnect.
Real fix: xce_test's `!<shell-command>` escape is handled in the single
top-level REPL loop before context-specific dispatch, so it works in any
context - including worker mode - without calling leave_ctx()/disconnecting.
inject_resources() now shells out via `!` to run the controller check
in-line, between resupdate and quit, while the worker's connection is still
open. query_controller_resvals()'s comparison logic is unchanged - this is
a pure sequencing fix. cpuf restored to DEFAULT_RESOURCE_ARGS as live proof.
Live-verified: cpuf now MATCHes at the controller layer (was always a false
MISMATCH before); adversarial shell-injection attempt via a malicious
resource name confirmed safe (printf %q quoting); SIGTERM delivered exactly
during the nested check confirmed zero orphaned/zombie processes across 4
independent runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_scheduler_log()/query_scheduler_resvals() tail the live scheduler
daemon's log (task 4.1's trace line) for each accepted injected resource,
with a bounded ~15s wait batched across all pending resources per poll
iteration (not N times the timeout). Reports MATCH / MISMATCH (both
values) / UNREACHABLE (timeout, distinct from mismatch), tracked in
SCHEDULER_OBSERVED/SCHEDULER_CHECK_RESULTS + scheduler-check-results.tsv.
Live-proven immune to bug-1579 (unlike the controller check): injecting a
host-attrs resource (cpuf) simultaneously produces a controller MISMATCH
and a scheduler MATCH, since this check reads task 4.1's trace log (written
once, at apply-time) rather than re-querying live state after the worker
disconnects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
query_controller_resvals()/controller_values_match() query the controller
via resval for every accepted (not rejected) injected resource/worker pair
and report MATCH/MISMATCH with both values on a mismatch, tracked in
CONTROLLER_OBSERVED/CONTROLLER_CHECK_RESULTS + controller-check-results.tsv
for tasks 6.3/6.4/7.1.
Also fixes bug-1579 (round-2, in default set only): cpuf, a host-attrs
resource the controller resets to baseline on worker disconnect, always
false-mismatched since inject_resources() disconnects before this check
runs. Swapped for r1m (periodic/dynamic, disconnect-safe) in
DEFAULT_RESOURCE_ARGS. Custom host-attrs resource=value args still hit this
- documented as a known follow-up, not fixed here (would need a C-level
xce_test change or a coprocess restructuring).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
inject_resources() joins a synthetic worker through xce_test's worker
context and runs resupdate for the requested resource=value pairs
(default: Requirement 1's numeric gap set + mem/cpuf regression names,
deliberately excluding type/model due to bug-1574's live segfault risk).
Per-name accept/reject results captured into INJECTED_VALUES/
INJECTED_RESULTS plus injection-results.tsv for tasks 6.2-6.4.
Also suppresses the real xcecd_virtual worker daemon that deploy-ha.sh's
pre-existing worker branch would otherwise silently start whenever the
build dir contains one (XCE_WORKER_BIN=/bin/true), which would have
violated Requirement 2.1 - bug-1577.
Blocked pending a separate prerequisite fix (bug-1578, commit b406dd7):
live testing surfaced a pre-existing controller crash on worker disconnect,
unrelated to this feature, now fixed with regression coverage. Live-
reconfirmed post-fix: controller survives the full join+resupdate+quit flow
indefinitely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sched_worker_change() aliased w->host.resvals_to_sched to a local xlist_t's
backing array, pushed it to the scheduler, then freed that array via
xlist_fini() without resetting the alias - leaving a dangling pointer for
the rest of the worker's life. On disconnect (handle_worker_leave ->
mark_worker_unavail -> persist_worker_hook_fun -> capnp_encode_worker ->
encode_Worker -> encode_NameValue_list), this dereferenced freed memory and
crashed the controller - but only once a scheduler had ever connected
(on_scheduler_join gates the hook registration), which made it easy to miss
in isolated worker-only testing.
The correct reset already existed in the function but was dead-code'd
behind #if 0 /*FIXME*/ - apparently from a prior fix attempt that got
wrapped instead of landed. Fix mirrors the proven-correct pattern already
used by the sibling apiserver_worker_change(): null the alias before
xlist_fini() frees the backing array, rather than double-freeing via
free_name_val_list() first.
Adds test_sched_worker_change.c, a regression test driving the real
production hook-dispatch path (xce_run_hooks -> sched_worker_hook_fun ->
sched_worker_dead -> sched_worker_change) and the real encode_Worker()
encoder the original crash traversed - confirmed to reproduce the exact
ASAN heap-use-after-free when the fix is reverted, and pass clean when
restored, closing the gap that let this regress silently once already.
Found while live-testing the resource-value-verification feature (deploying
a scheduler + injecting worker resource values via xce_test, then
disconnecting) - unrelated to that feature's own changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run-resource-verify.sh deploys controller+scheduler(-vvv)+apiserver(debug
logging) via xce-deploy-ha's deploy-ha.sh -> start-all.sh into a fresh
scratch dir, confirms all three are genuinely up, and cleans up via a trap
registered before any process starts - covering both normal completion and
a mid-deployment abort. Logs always preserved regardless of --keep-running.
Independently stress-tested with 8+ SIGTERM interrupts at varying
deployment points and a genuine crashing-apiserver scenario - zero leftover
processes in every case. Single-node by default (propagation testing, not
HA failover), overridable via $XCE_RESVERIFY_PEERS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One logger.Debug("resource-location stored", ...) call in
OnResourceLocationUpdated, immediately after store.ResourceLocations.Put,
logging resource/host/value as zap fields. Debug-level only, matches this
package's existing logging convention (reuses convert.go's field names).
Live-confirmed against a real deployment with logging.level: debug.
Also surfaced (but did not fix, out of boundary - pure Go change, no
possible causal link) a pre-existing xce_test segfault on string-typed
resupdate (type/model) - uninitialized stack memory, non-deterministic,
invisible to ASan. Logged as bug-1574; flagged in tasks.md as a risk for
task 7.1's default resource set and task 8.1's live e2e run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One TRACE (tlog_debug3) line in xce_core_push_events()'s RESVALS_DELTA case,
logging host/resource/value in a fixed format ("resval applied host=...
resource=... value=..."), immediately after the value is written to the
worker's live state. Debug-level only, no control-flow change.
Explicit svalue/fvalue branching (round-2 fix after review) - never reads a
Value_u_t union member without confirming .which matches first, since
Value_u_bvalue (boolean resources) is reachable at this path in production
and would otherwise be misread as a garbage float.
Live-confirmed against a real deployment at -vvv. Also surfaced (but did not
fix, out of boundary) a pre-existing double-free crash on a resource's
second periodic report - logged as bug-1572.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a
pre-built /tmp/apiserver binary, not built by this script) that generates
start-apiserver.sh with a debug-level logger.yaml, plus an always-generated
start-all.sh sequencing controller-health-check-then-worker/scheduler
(-vvv)/apiserver. Purely additive - callers without a built apiserver
(e.g. xce-test-ha) see zero behavior change.
Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the
step; with-apiserver reaches full backend registration with debug logging
active and scheduler emitting DEBUG3 output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cmd_resupdate() now issues a follow-up xcecli_resval() query per submitted
(resource, worker) pair after the update RPC, since update_resval_list()'s
reply can't carry per-item status. Reports accepted/rejected per name, with
a bounded retry (5x100ms) against propagation races.
Live-confirmed: `resupdate -n mem -v 5 -n not-a-real-resource -v 1` reports
mem accepted and not-a-real-resource rejected in the same invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
set_resval() now delegates type resolution to resval_typeresolve.c; unknown
resource names default to numeric, overridable via a new -t <numeric|string>
flag. cmd_resupdate()'s getopt loop defers per-pair finalization so a
trailing -t still applies to its pair, while preserving the exact original
duplicate-value rejection behavior (round-2 fix after review caught a
last-write-wins regression in that refactor).
Live-confirmed: resupdate -n r15m -v 1.5 (previously silently dropped) now
succeeds and is visible via resval.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Documentation-only stub: usage, VERDICT line format, and PASS/FAIL/cleanup
contract per design.md, explicitly marked not-yet-functional. The actual
run-resource-verify.sh orchestration script is built incrementally by
tasks 3.1-7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move set_resval()'s resource-name lookup table into a new, dependency-light
resval_typeresolve.c/.h module so it's independently unit-testable (test.c
can't be white-box included into Unity). Adds RED tests for the not-yet-
implemented numeric-fallback/type-override behavior (task 2.1).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
