apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/
CKPT/PREASON dispatch case only ever called the generic, state-only
apply_job_update_transition(j, state) -- start_time/finish_time/run_time
were never touched, so `xcecli joblist` always showed StartTime=0 for
RUN/DONE/EXIT jobs and FinishTime=0 for DONE jobs (EXIT jobs got a
finish_time only via the separate, richer WAL_JOB_FAIL dispatch path).
Fix is additive: stamp time(NULL) keyed on wal_type right after
apply_job_update_transition() succeeds, still under the same
lock(j, RTTI_LOCK_WRITE) -- WAL_JOB_START sets start_time, WAL_JOB_FINISH
sets finish_time and derives run_time. No change to
apply_job_update_transition()'s signature or the dispatch structure.
Add test_ace_job_start_sets_start_time and
test_ace_job_finish_sets_finish_time_and_run_time to
test_apply_committed_entry.c, following the existing tace_seed_job
fixture pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
makeGroup()'s existing-group update path (hit whenever a config-push
re-applies a group name that already has members, e.g. a retried config
push during controller failover) released xce_g_member_t.u.name via
FREEUP() (plain free()). u.name is allocated by makeString(), an
rtti-managed, refcounted pointer that is not a malloc()-base address --
freeing it directly corrupts the heap. group_killer() in the same file
already releases the identical field correctly via dereference().
Switch both FREEUP(m->u.name) call sites to dereference(m->u.name) to
match. Add a regression test that re-applies config for the same group
name twice, reproducing the exact ASAN bad-free at group.c:234 pre-fix.
This is the root cause of the reproducible xce_scheduler
"free(): invalid pointer" SIGABRT crashes seen during 3-attempt
controller-failover config-push retries (8 confirmed core dumps).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers the printf/tlog-prefix consistency fix, the "info resource all"
UX fix, and the resource-distribution-mapping roadmap gap surfaced while
investigating total/current display for GLOBAL/SHARED resources.
Surfaced while investigating a user request to show total/current values
for GLOBAL/SHARED resources in the scheduler REPL's "info resource"
command -- researched and confirmed that concept doesn't exist anywhere
in this codebase (resourcemap config field dropped at build time, capnp
fields commented out, worker.c actively rejects XCE_RESF_SHARED resources
today). This spec already owns closing that gap; it just never made it
into the roadmap's backlog list.
Correct filtering (bug-1593: only XCE_RESF_GLOBAL/XCE_RESF_SHARED
resources list under "all") but confusing UX: a session with only
host-based resources (e.g. just "slots") produced zero output lines,
indistinguishable from a REPL/dispatch bug. Now prints an explicit
"no global/shared resources defined" message when the count is zero.
xce_scheduler link-wraps printf (-Wl,--wrap=printf, src/meson.build's
wrap_link_args) to route every plain printf() call through tlog, stamping
the full "[timestamp] [pid] [LEVEL] [xce_vprintf]" prefix. sim_print_decision()
(sim_loop.c) used plain printf() for its DISPATCH/PEND/PREEMPT lines, so
those lines carried that prefix while every sibling "[sim] ..." line in
the same REPL/sim output (event echoes, breakpoint hits, banners, info
command output -- all fprintf(stdout, ...)) stayed prefix-free. Found via
a live deployment log. Switched to fprintf(stdout, ...), matching the
established convention everywhere else in this codebase.
Invisible to the entire prior test suite for a structural reason: only
the real xce_scheduler executable links with wrap_link_args, not the
white-box unit-test binaries, so printf() and fprintf(stdout, ...) were
indistinguishable inside any unit test. Added a mechanical regression
guard instead of a runtime test: test_no_bare_printf.c source-scans every
src/scheduler/src/*.c and src/scheduler/src/plugins/*.c file for a bare
printf( call (same source-text-scan genre as test_core_isolation.c).
Covers the low-risk backlog cleanup pass (stale roadmap entries,
job_ctrl/OP_upgradeConfig investigations) plus one self-correction lesson
in cerebrum.md's Do-Not-Repeat list.
- xce_test shell command support: was already implemented (xce-test-e2e-
primitives task 2.1, closed 2026-06-19) -- stale "not started" entry
corrected.
- scheduler-core tasks 2.1-2.6 _Verified: backfill: was already done
(commit d7b49690, 2026-08-08) -- corrects a status line I myself wrote
incorrectly earlier today by trusting the original 2026-08-07 finding's
wording without re-checking the file.
- job_ctrl direct-mutate-path investigation: completed. Real risk
confirmed (pre-commit state pushed live to scheduler + apiserver/clients
with no rollback on commit failure), scope wider than originally framed
(on_job_stop/on_job_resume/on_job_ckpt share the same pattern, fed by
ordinary traffic not just rare admin commands), plus a separate
read-lock-during-mutation bug found along the way. Not promoted to a
spec yet -- scope/framing corrected first per this session's decision.
- OP_upgradeConfig standby redirect verification: investigated. The
item's own premise doesn't match production (a standby never actually
returns a redirect for this opcode -- a documented, deliberate feature
gap, not a missing test); coverage confirmed already complete on both
the client mock side and a live 3-node regression test.
design.md's own "Open Questions/Risks" list (items 5 and 6) still described
the OPCode_schedAdmin admission gap and the tlog activation gap as open/
deferred, even though both were fixed earlier in this session -- marked
FIXED with pointers to the actual change. research.md's schedAdmin
decision record and its Risks & Mitigations bullet got the same treatment
(SUPERSEDES note, historical record preserved rather than rewritten).
tasks.md's 2026-08-07 GO/NO-GO follow-up list gained a status-update line:
follow-up 1 (tlog) is now fixed, follow-up 2 (fcfs.c/builtin_fcfs.c
traceability) is only partially closed (design.md text was already
correct; tasks 2.1-2.6 still lack _Verified: lines -- flagged, not
fixed here since I have no fresh verification evidence for those tasks).
roadmap.md's scheduler-core summary line corrected to attribute each gap
to the right source (2026-08-07 validate-impl vs today's separate
design-review pass) instead of conflating them.
Task 12.5 (REPL live-operator-testing fixes) was implemented and its bugs
fixed in an earlier commit, but the tasks.md checkbox was never flipped and
carried no _Verified: line, unlike every other completed task in this spec.
Ran /kiro-verify-completion against it with fresh evidence rather than
trusting the earlier self-report: fresh build+full suite (41/42, only the
pre-existing unrelated bug-1560 failing), a live subprocess smoke test of
the real xce_scheduler binary for all five amendment gaps (including a
hand-built .replay fixture with resource data to close the one gap the
first pass left unit-tested-only), and direct source inspection of each
fix. All five confirmed correct; checkbox flipped, _Verified: line added.
Two retrospective design-review findings:
- design.md's System Flows sequence diagram showed the controller-layer
resval check as an independently-launched client session, implying no
dependency on the injecting worker staying connected. As-built (bug-1579),
host-attrs resources reset to baseline the instant the worker disconnects,
so the check actually runs nested inside the still-open worker session via
xce_test's `!` shell-escape. Updated the diagram and prose to show this,
and documented the still-open residual risk for custom host-attrs
resource names in Error Handling.
- Added explicit "do not change this format" warning comments at both
trace-log call sites (core_events.c's RESVALS_DELTA case, apiserver's
OnResourceLocationUpdated) that xce-test-resources' log-grep depends on --
an unrelated logging cleanup at either site would otherwise silently turn
every future verification run into a false UNREACHABLE/timeout.
No behavior change in either source file -- comments only.
The schedAdmin dispatch case blocks its vtask handler thread for up to 5s
waiting on a cross-process reply while holding c->sched.lock -- bounded
blocking on a vtask thread, which ha-async-io.md permits only under an
admission reservation. The real risk isn't just the wait itself: since
c->sched.lock serializes concurrent requests one at a time, a burst of N
concurrent schedAdmin calls could already occupy N vtask threads for up to
N*5s each before this fix.
Now acquires vtaskmgr_blocking_acquire(&c->vtaskmgr) before ever locking
c->sched.lock, releasing on every exit path -- the same shared admission
budget commit.c's commit-wait barrier already gates on (wired via
commit_set_pool). Admission-exhausted requests fast-reject with
{ok=false, error="server busy, try again"} rather than queuing. New test:
test_schedadmin_fast_rejects_when_admission_exhausted, using a poison
c->sched.ep sentinel to prove the fast-reject happens before any
lock/forward attempt.
test_core_isolation.c's build-time proof (nm(1) undefined-symbol scan of
sched_core_isolation_lib) previously covered only core.c -- core.c's own
800-line-cap siblings, holding scheduling logic just as boundary-sensitive
as core.c itself, were "isolation-clean by review" only, with nothing
mechanical catching a regression. Extends sched_core_isolation_lib to
compile all four files together, and the source-text scans (forbidden
headers/symbols, wall-clock calls) to also cover core_events.c/
core_session.c. core_clock.c joins only the nm(1) layer, not the
wall-clock text scan -- it's the one sanctioned exception carrying the
core family's single legitimate clock_gettime() call.
sched_sim_driver_start() now registers the core's tlog format function and
marks its core active for logging right after the virtual clock is seeded
(xce_core_register_tlog_format()/xce_core_set_active_for_logging()) --
every log line for the rest of the run shows frozen/scaled replay time
instead of real wall-clock time, matching what an operator debugging
against a replay's own timeline needs. sched_sim_driver_run() deactivates
on all 7 of its return paths, since there's no sim-driver teardown
function and g_active_core_for_logging would otherwise dangle past the
caller's stack frame.
Also fixes a genuine pre-existing ASAN stack-use-after-return (bug-1598)
the new deactivation test exposed: sim_picker_thread_fn is spawned but, by
design, never joined/cancelled, so a stack-local `xce_sched_sim_driver_t
sim` in test_sim_event_loop.c was always a latent UAF once its owning test
function returned -- just not previously triggered by any prior test
ordering. Switched the three affected fixtures in that file to `static`
storage (safe: sched_sim_driver_start() memsets the whole struct on entry,
and none of the three call sites are re-entrant).
Adds auto-detection of real vs virtual worker based on systemd
availability (/run/systemd/system), plus an XCE_REAL_WORKER override.
Real xcecd requires root (systemd D-Bus transient units, hardcoded
/var/run paths) and is used opportunistically now that passwordless sudo
is available in the sandbox; defaults to xcecd_virtual when systemd/D-Bus
isn't available.
Bugs fixed, discovered across a day of live-operator testing against a
real deployed cluster:
- bug-1584: clock anchors never initialized to real wall-clock values,
timestamps showed system uptime instead of Unix epoch.
- bug-1585: heap-use-after-free in the normal-mode driver's periodic
resval update path -- core_events.c's RESVALS_FULL/_DELTA handlers do a
shallow zarray_put() of the wire-decoded xce_val_t*, but driver.c's
release_translated_payload() unconditionally freed the same decoded
array right after, so the next periodic report for the same
worker+resource triggered a UAF read via kill_sched_resval(). New
regression test: test_resval_ownership.c.
- bug-1587/1588/1591: raw-ns/raw-uint64 jobid/timestamp prints across the
REPL/SIM family converted to the same human-readable formatting already
used elsewhere (fmt_jobid(), timestamp formatting).
- bug-1590: repl_info.c's XCE_VAL_NUMERIC checks corrected to
Value_u_fvalue, fixing `info worker`/`info resource` falling back to
0.00 for real numeric resvals.
task 12.5 (Req 5.22 amendment) -- five REPL gaps surfaced by a live
operator session, each with its own bug id:
- bug-1594: `continue`/`step` reaching EOF with no breakpoint hit now
re-enters the REPL via xce_repl_on_replay_complete() instead of exiting
the process; step/continue issued after replay is already complete
re-prompt instead of silently exiting.
- bug-1595: `step`/`continue` now echo each input event (not just
decisions) via repl_print_event_summary(), in real event/decision
order.
- bug-1593: `info resource <name>|all` now reports resource definition
metadata restricted to XCE_RESF_GLOBAL/XCE_RESF_SHARED resources, read
from core->sharedInst, instead of aggregating host-based resources
across every worker.
- bug-1596: repl_split_tokens() strips one matching pair of surrounding
quote characters from each token.
See requirements.md's Req 5 AC 22 "Amendment (2026-08-19)" for the full
rationale on each of the five task 12.5 gaps.
Full scheduler suite green (41/42; the one remaining failure,
test_recordstart_symlink_escape_replies_error, is the pre-existing
unrelated bug-1560).
fmt_jobid() renders the raw idgen()-packed jobid in the same
cluster-apiserver-timestamp-sequence form already used everywhere else
this codebase logs/displays a jobid (scheduler.c/handler.c, worker's
process_manager.c, xcecli, waldump). job.c's own TRACE lines were the one
place still printing the raw 64-bit packed integer.
Every early-exit branch in the worker_list scan either re-tested the same
node forever (bare `continue`) or returned out of the whole function on the
first DONE/EXITED worker, so job array elements past the head of the list
never got their on_job_start report. Capture `next` before touching each
node and always advance to it. Affects xcecd_virtual and real xcecd alike
(same DBUS_MANAGER-gated function).
Also records a roadmap follow-up (xcecd_virtual concurrent-job support,
MXJ > 1) discovered while investigating this bug.
requirements.md, design.md, research.md, and spec.json were never committed
during implementation (only tasks.md was, per the per-task checkpoint
convention). Committing them now that all 15 tasks are implemented,
reviewed, and validated GO by /kiro-validate-impl.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
verdict-logic-selftest.sh sources the real aggregate_verdict()/
build_verdict_record() (via run-resource-verify.sh's BASH_SOURCE guard, no
live deployment triggered) and exercises them with synthetic per-layer
records: all-match/PASS positive control, a single-resource single-layer
MISMATCH within a multi-resource batch (asserting both correct attribution
of the failure AND that the other resources independently still report
PASS), UNREACHABLE distinct from MISMATCH, REJECTED, and a zero-records
degenerate case. 53 assertions total.
Proven non-tautological: breaking aggregate_verdict()'s comparison logic
(two different ways, independently reproduced by review) makes the selftest
genuinely fail; restoring makes it pass again.
This completes the resource-value-verification spec - all 15 tasks (1.1
through 8.2) implemented, reviewed, and committed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Validation-only, no code changes. Confirmed the complete spec end-to-end
against a freshly built controller+scheduler+apiserver: default 9-resource
set + one deliberately-unregistered name -> VERDICT resources=10
rejected=1 result=FAIL (exit 1), attributable specifically to the rejected
name, not a false PASS; positive control (defaults only) -> VERDICT
resources=9 rejected=0 result=PASS (exit 0). All 27 individual layer-checks
(9 resources x 3 layers) MATCH; raw scheduler.log/apiserver.log trace lines
independently verified against tasks 4.1/4.2's documented format; rejection
confirmed non-silent across 4 independent log sources; cleanup and log
preservation confirmed.
Surfaced (but correctly did not fix, out of boundary) bug-1583: the
apiserver stores resource-location updates with zero name validation, even
for names the controller/scheduler both reject. Confirmed this does not
affect this feature's own correctness - accept/reject is determined
entirely by cmd_resupdate()'s controller-side follow-up query, before any
layer is checked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
aggregate_verdict()/build_verdict_record() combine the three per-layer
check results into a single per-resource comparison, self-contained and
invokable with synthetic records (task 8.2). Uses ASCII Unit Separator
(\x1f) as the record delimiter after discovering bash's IFS tab-splitting
silently collapses adjacent empty fields even with IFS set to only a tab
(bug-1581), which corrupted UNREACHABLE-record field alignment.
Emits one machine-parseable VERDICT line (resources/layers_checked/
mismatches/unreachable/rejected/result) plus the terminal PASS/FAIL oracle
line and matching exit code. Cleanup is the same trap-based mechanism task
5.1 registered for the abort path - logs always preserved regardless of
--keep-running.
Round-2 fix: main()'s live-data wiring (build_verdict_records_from_tsv())
derived a layer's status from array-key existence rather than the actual
recorded outcome - since tasks 6.3/6.4 always set the key (even on a
genuine timeout, just with an empty value), every real UNREACHABLE was
silently misclassified as MISMATCH. Fixed to check SCHEDULER_CHECK_RESULTS/
APISERVER_CHECK_RESULTS string content instead (bug-1582). Live-confirmed
by killing the scheduler mid-run: VERDICT now correctly reports
unreachable=9 mismatches=0, not the reverse.
SKILL.md rewritten to describe the now-complete, functional tool.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_apiserver_log()/query_apiserver_resvals() mirror task 6.3's
scheduler-check pattern: bounded-wait polling (batched across all pending
resources per iteration) of apiserver.log for task 4.2's "resource-location
stored" trace line, parsing the trailing zap JSON block. Reports MATCH /
MISMATCH (both values) / UNREACHABLE (timeout, distinct from mismatch),
tracked in APISERVER_OBSERVED/APISERVER_CHECK_RESULTS +
apiserver-check-results.tsv.
This completes all three per-layer checks. Live-confirmed: the full default
resource set (including cpuf, post-bug-1579-fix) MATCHes simultaneously at
the controller, scheduler, and apiserver layers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously worked around (task 6.2) by removing the one host-attrs resource
(cpuf) from the default set - a custom resource=value arg using any
host-attrs name would still false-mismatch at the controller layer, since
inject_resources() disconnected the worker before the separate controller-
check session ran, and the controller resets host-attrs resources on
disconnect.
Real fix: xce_test's `!<shell-command>` escape is handled in the single
top-level REPL loop before context-specific dispatch, so it works in any
context - including worker mode - without calling leave_ctx()/disconnecting.
inject_resources() now shells out via `!` to run the controller check
in-line, between resupdate and quit, while the worker's connection is still
open. query_controller_resvals()'s comparison logic is unchanged - this is
a pure sequencing fix. cpuf restored to DEFAULT_RESOURCE_ARGS as live proof.
Live-verified: cpuf now MATCHes at the controller layer (was always a false
MISMATCH before); adversarial shell-injection attempt via a malicious
resource name confirmed safe (printf %q quoting); SIGTERM delivered exactly
during the nested check confirmed zero orphaned/zombie processes across 4
independent runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_scheduler_log()/query_scheduler_resvals() tail the live scheduler
daemon's log (task 4.1's trace line) for each accepted injected resource,
with a bounded ~15s wait batched across all pending resources per poll
iteration (not N times the timeout). Reports MATCH / MISMATCH (both
values) / UNREACHABLE (timeout, distinct from mismatch), tracked in
SCHEDULER_OBSERVED/SCHEDULER_CHECK_RESULTS + scheduler-check-results.tsv.
Live-proven immune to bug-1579 (unlike the controller check): injecting a
host-attrs resource (cpuf) simultaneously produces a controller MISMATCH
and a scheduler MATCH, since this check reads task 4.1's trace log (written
once, at apply-time) rather than re-querying live state after the worker
disconnects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
query_controller_resvals()/controller_values_match() query the controller
via resval for every accepted (not rejected) injected resource/worker pair
and report MATCH/MISMATCH with both values on a mismatch, tracked in
CONTROLLER_OBSERVED/CONTROLLER_CHECK_RESULTS + controller-check-results.tsv
for tasks 6.3/6.4/7.1.
Also fixes bug-1579 (round-2, in default set only): cpuf, a host-attrs
resource the controller resets to baseline on worker disconnect, always
false-mismatched since inject_resources() disconnects before this check
runs. Swapped for r1m (periodic/dynamic, disconnect-safe) in
DEFAULT_RESOURCE_ARGS. Custom host-attrs resource=value args still hit this
- documented as a known follow-up, not fixed here (would need a C-level
xce_test change or a coprocess restructuring).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
