Extend the IS_WORKER_UNAVAIL-style candidate-host filter to also skip
a host with XCE_WORKER_RECONCILING set (Req 5.1), naturally
re-including it once the bit clears with no separate mechanism (Req
5.2). Leaves the scheduler's decision-publish code and its
symbol-guard regression test untouched.
Fixes bug-1751: design.md's cited filter site (scheduler.c's
sched_reset_worker_(), via sched_begin_session()) is dead code on both
live drivers - driver.c/sim_loop.c actually call
xce_core_begin_session() -> core_session.c's core_reset_workers(), an
independently-maintained mirror not listed in design.md's File
Structure Plan. Applied the identical disjunct to both sites so the
task's own observable-completion criterion holds in production, not
just on the literally-cited dead path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add OPCode_jobStateQuery dispatch to myreq_handler with multi-page query
reassembly, routing to task 4.2's xcecd_resync_answer_query() (Req
8.1, 8.3). Gate the worker's resumption of normal post-join operation
on a resync_pending flag until the full report has sent (Req 3.8).
Fixes bug-1749 (xcecd_resync.c was never added to the worker's
production source lists - task 4.2's build only ever exercised it via
the test binary's white-box include).
Round 1 review caught bug-1750: the gate sat as checking_connection()'s
entry guard, so a connection drop mid multi-page query left
resync_pending stuck forever, permanently disabling the worker's only
reconnect mechanism (Req 6.1 violation). Round 2 narrows the gate to
guard only the rejoin action, and resets it from xcecd_on_reconnect()
so a stalled query can never wedge it past one reconnect cycle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Classify each queried job ID as running/finished-ok/finished-fail/
unrecognized (Req 3.1-3.3, 3.6), report unqueried running/recently-
finished jobs as extras, and send both via task 1.4's paginated wire
helper (Req 8.2, 8.3). Composes correctly with the already-implemented
controller-side consumer (resync_apply_report).
Adds xcecd_finished_job_retention_snapshot() to xcecd_worker_utils, a
small additive companion needed to enumerate all retained jobs for the
extras set (task 4.1 only exposed single-jobid lookup).
Folds the still-open OPCode_jobStateQuery receive-dispatch wiring gap
into task 4.3's scope in tasks.md, since 4.3 already needs the real
receive path to implement its post-join gate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Retain a job's exit status/code/true finish time for a configurable
window after update_worker_status() detects its exit, before
cleanup_worker_list() would otherwise discard all trace of it
(Requirements 3.4, 3.5, 3.7). Hooked into the single choke point both
production exit-detection paths (libuv IPC callback, dbus polling)
converge through, so retention happens-before any reap regardless of
which mechanism is active.
Also prunes on retain (not just on lookup), closing a review finding:
lookup currently has zero production callers until task 4.2 lands, so
without this the store would grow unboundedly for the daemon's uptime.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Task 4.1 discovered and correctly self-scoped away from a real gap: the
worker daemon has no delivery path at all today for any ClusterConfig
field -- worker_push_config's existing wire push carries a generic
confvars key/value list plus a small number of directly-pulled fields
(clustername is the existing precedent). Confirmed by direct trace
(controller/worker.c's collect_confvars/conf4worker construction, and
grep of worker/src/controller_connector.c's opcode dispatch switch,
which has no clusterconf case). Added workerResyncWindowMs delivery to
task 6.1's scope, mirroring the clustername precedent exactly -- no new
opcode needed, reuses the existing OPCode_push/WorkerConfig struct that
task already touches (on_worker_join). Without this, task 4.1's
retention window (built correctly, as a settable seam with a documented
placeholder default) would never actually receive an operator-tuned
value in production.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add resync_apply_report()/resync_apply_running_entry()/
resync_apply_finished_entry()/resync_apply_unrecognized_entry() to
resync.c: routes each entry in a worker's job-state report to the
correct existing idempotent path (still-running -> clear from outstanding
set, no fact; finished-success/failure -> record_job_finish_fact_at/
on_job_fail; unrecognized -> record_job_reconcile_pending_fact;
unqueried extra -> record_job_reconcile_assign_fact or no-op), discards
reports for superseded rounds (Req 6.1/6.3), and resolves the round once
its outstanding set empties.
Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(),
additive alongside the unmodified record_job_finish_fact, so a
finished-success reconciliation entry records the WORKER-reported
finish time instead of reconciliation-processing time -- proven via a
real WAL-write + persist_recover() restart-replay test. The FAIL-side
half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at
all, for any caller) is confirmed pre-existing and systemic, correctly
left out of this task's scope.
Three review rounds surfaced and fixed two real, distinct bugs in the
unqueried-extra-job routing guard (see tasks.md's Implementation Notes
for the full account and the general lesson about re-deriving a full
truth table after a partial boolean-guard fix): round 1 caught the
wrong axis (terminal-vs-non-terminal instead of assignment-status,
Req 4.8); round 2 caught that the first fix's short-circuit clobbered
an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including
confirming the implementer's own test had certified the bug as correct;
round 3 independently re-derived the final three-way guard from scratch
and found no further defect. bug-1748 logged for the guard fix.
Full controller suite: 1239 tests, 0 failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
New src/controller/resync.c/resync.h: resync_arm_round(cluster_t*,
worker_t*) enumerates every non-terminal job assigned to a host,
populates worker_t.resync (round_token, query job-ID set, deadline),
sets XCE_WORKER_RECONCILING, and schedules a non-blocking delaymgr
one-shot timeout. resync_resolve_round() clears the round; a fired
timeout resolves it too, via resync_round_is_current()'s by-value
round-token check -- delay.c/delay.h expose no true cancel/dequeue
primitive, so a stale timer from an already-superseded round becomes a
safe no-op instead of touching a newer round's live state (verified
safe against both misresolving the wrong round and use-after-free: the
timeout closure never holds a pointer into resync_state_t itself).
5 new tests: query-set correctness, fresh-token-per-arm, the
RECONCILING flag's set/clear lifecycle, and a direct (non-blocking)
timeout-fire proof via an XCE_TESTING-gated seam confirmed compiled out
of the production binary. Full suite: 1232 tests, 0 failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Split dispatch_a_job_()'s combined "(w->ep == NULL) || (w->host.status
!= 0)" check into two branches, checked in order: w->ep == NULL now
resolves directly via record_job_reconcile_pending_fact() (task 2.1) --
no dispatch_fail_count increment, no mark_job_fail(), never reaching
xce_classify_dispatch_fail() at all -- before the code ever tests
w->host.status != 0, which keeps its existing retry-then-terminal-fail
behavior completely unchanged (byte-identical body, confirmed against
HEAD).
Completes Task 2 (all controller job-table reconciliation outcomes:
2.1 reconcile-to-pending, 2.2 reconcile-assign, 2.3 this split).
RED phase caught a genuine pre-existing test bug: test_worker_notready_
resolves_retryable was seeding a NULL-ep worker while claiming to test
the host.status!=0 retry path -- it never actually exercised that path.
Corrected to use a connected-but-busy worker, with self-asserting
TEST_ASSERT_NOT_NULL(w->ep)/TEST_ASSERT_TRUE(status!=0) guards so this
can't silently drift again.
2 new tests, including a full JOB_TRANSITION_APPLIED path proof (WAL
LSN advance, hook-fires-exactly-once per bug-1679, allocs released per
bug-1631) through this exact call site, both using correctly-timed
canary references on both the job and worker objects to catch any
over-release under ASAN. Full controller suite: 1227 tests, 0 failures,
zero regressions across every other dispatch_a_job_ failure site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*,
xce_job_state_entry_t*) to job.c: eagerly sets job->job.state to match
the worker's reported outcome (mirroring dispatch_a_job_'s inline
mutation, minus allocation fields -- there are none to set, job->job.allocs
stays NULL per Requirement 4.8's documented scope). Calls the existing
persist_submit_job_dispatch_fact(..., alloc=NULL) directly, then leaves
the post-unlock persist_notify_job_updated_already_persisted() call to
the caller, same convention as task 2.1.
Fixes bug-1746 along the way: capnp_encode_joballoc() (persist.c, part
of bug-1678's original fix, predates this feature) unconditionally
dereferenced a NULL alloc -- never triggered before since
dispatch_a_job_ (the only prior caller) always passes non-NULL. Fixed
with a minimal guard producing a true zero-length buffer, matching
apply_committed_entry's alloc_len > 0 decode gate exactly. This
directly contradicted design.md's "tested, backward-compatible on both
seams" claim (true only for decode, never actually exercised for
encode) -- corrected in 4 places.
2 tests: applies-running-no-allocs-and-notifies, and a real restart-
replay round trip (write to real disk, fresh persist_recover(), confirm
jobid/host/state/NULL-allocs survive) -- proving this task's own
"durably survives a restart replay" completion criterion literally, not
just the in-memory mutation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Found live during task 2.2 implementation (bug-1746): design.md claimed
persist_submit_job_dispatch_fact's NULL-alloc case was "explicit,
tested, backward-compatible" on both apply seams -- true for the DECODE
side (apply_committed_entry/apply_wal_entry, bug-1678's actual fix
target), but the ENCODE side (capnp_encode_joballoc) was never
exercised with NULL before this feature and had a latent unconditional
NULL dereference. Corrected all references to distinguish encode vs.
decode safety and note the bug-1746 fix task 2.2 required.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add record_job_reconcile_pending_fact(cluster_t*, job_t*) to job.c:
eagerly applies apply_job_update_transition(job, JOB_STAT_PEND) (real,
live mutation, not a deferred peek -- matching dispatch_a_job_'s
established pattern, not record_job_finish_fact's), releases
job->job.allocs on an applied (non-no-op) transition, then durably
appends via the generic persist_submit_job_fact(WAL_JOB_UPDATE). Caller
holds job's lock throughout and is responsible for calling
persist_notify_job_updated_already_persisted() after unlocking -- this
function deliberately does not call it itself.
Closes bug-1631 at this new call site: a RUN job with a real allocation
reconciled to PEND (Requirement 4.6, worker reports it unrecognized)
would otherwise carry a stale allocation into its next dispatch attempt,
exactly the gap bug-1631 documented as dormant until something re-admits
an already-dispatched job to the scheduler's PEND pool -- this feature
is that path. dispatch_a_job_ itself is unchanged (still correct for a
job's first-ever dispatch, the only case reachable there).
3 tests: allocs release + eager-mutation-before-unlock proof, no-op on
already-pending, and the Requirement 4.4 terminal-state-preservation
proof (rejects overwriting JOB_STAT_DONE).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
All 5 sub-tasks (1.1-1.5) done: wire protocol schema, worker reconciling
status type, worker test infra (pre-existing), wire codec helpers, and
xceadmin config-build wiring.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add workerResyncWindowMs/workerResyncTimeoutMs to xce.schema.json
(minimum:0, matching the optional-scalar-field precedent -- these
aren't `required` with a floor like heartbeatInterval, so minimum:1
would be wrong) and to mapping.py's YAML->capnp field table. Regenerate
both golden config-build fixtures (task 1.1 grew ClusterConfig's wire
layout by 2 fields; full/expected.bin and minimal/expected.bin each
grew by exactly 8 bytes). full's fixture YAML now sets real values
(900000ms/15000ms); minimal deliberately left untouched (no xce.yaml)
per its own locked-down "hosts+queues only" boundary test -- its
expected.bin still regenerates to reflect the new default-0 fields.
Two new tests assert specific decoded values, not just "build succeeds".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add xce_job_state_query_req()/xce_job_state_report_req() to request.c,
paginating job IDs / job-state entries across OPCode_jobStateQuery /
OPCode_jobStateReport messages via direct Request.body union arms
(mirroring xce_jobdispatch_req/xce_notify_req's direct-send shape, not
xce_push_data_req's XceData-table multiplexing -- these are dedicated
union arms, not multiplexed push/reply-get traffic). 8 round-trip tests
(normal/empty/multi-page for both query and report, covering all 4
JobStateStatus values and both queried states) all pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Discovered live during task 1.4 development: xceadmin's controller.capnp
copy is freshly synced from src/schema/ on every /xce-build xceadmin run
(meson-build.sh), so task 1.1's two new ClusterConfig fields already
broke the checked-in golden-fixture tests (test_golden_fixture_matches_
checked_in_expected_bin[minimal/full]) by changing the binary's wire
layout. Deeper gap: neither field has a YAML->binary configuration path
at all (unlike their own cited precedent, heartbeatInterval/
inactiveHeartbeatCount, which is wired into both xce.schema.json and
mapping.py) -- the original design review never scoped xceadmin as a
boundary. Added design.md Boundary Commitments addendum + tasks.md
task 1.5 to close both gaps together.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Worker-side C test infrastructure (src/worker/testsuite/, 5 passing
tests, meson suite:'worker' registration) already exists, built by
component-failover-adoption (2026-08-11), weeks before this spec was
written -- task 1.3's "no testsuite exists" premise was sourced from
stale steering docs (structure.md, testing.md, build-deployment.md).
Confirmed directly: fresh build + meson test --suite worker, 5/5 OK.
No source changes needed. Recorded the correct convention task 4.2 must
follow (per-file Unity executable + explicit test() registration, not
a controller-style aggregator runner) in tasks.md's Implementation Notes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add XCE_WORKER_RECONCILING status bit (0x00800000, next in the existing
sequential bitmask) + IS_WORKER_RECONCILING macro mirroring
IS_WORKER_UNAVAIL's shape, and resync_state_t (round_token,
queried_jobids array+count, deadline) referenced via a nullable
worker_t.resync pointer -- NULL when no reconciliation round is
outstanding. Compiles and is reachable, unused by any caller yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add OPCode_jobStateQuery/OPCode_jobStateReport, JobStateEntry/
JobStateQuery/JobStateReport structs (paginated, mirroring JobData/
WorkerData/ResourceVals' {ntotal,pos,count} shape), and
workerResyncWindowMs/workerResyncTimeoutMs ClusterConfig fields to
controller.capnp, plus the companion typemap.h/clusterconfig.h C type
declarations this codebase's capnp $C.mapname codegen requires to
compile. Both new opcodes are fire-and-forget (Request.body union only,
no RequestReply arm), matching design.md and the push/notify precedent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Design validation (2026-09-01) found and fixed 4 issues: a stale bug-1678
citation blocking reuse of persist_submit_job_dispatch_fact, missing
XCE_WORKER_RECONCILING accounting against scheduler-dispatch-commit's
dispatch-fail retry/terminal-fail classification, bug-1631 reactivation
(job.allocs never released on RUN->PEND reconciliation), and an eager-vs-
deferred mutation inconsistency between the new job.c functions and the
bug-1679 explicit-notify precedent. design.approved and tasks.approved
both set; tasks.md drafted (7 major/19 sub-tasks) and passed two rounds
of independent task-graph sanity review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
scheduler-dispatch-commit's 5 parent tasks were already fully implemented
(all subtasks checked); this rolls up the parent checkboxes to match.
commit-batch-window is a new spec, requirements-only, explicitly parked
(not actively pursued) pending a dedicated future session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
sched_reconcile_job_resource_usage() was only reachable from
sched_apply_job_rusage() (scheduler.c's sched_update(), which has no live
caller), so real-usage reconciliation never actually ran outside tests.
Wire it into core_events.c's XCE_CORE_EVENT_JOB_RUSAGE handler instead,
the actual live path shared by both normal-mode (driver.c) and sim/replay
mode (sim_loop.c). Adds link-satisfying stubs to the testsuite files that
#include core_events.c directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Resource information for candidate hosts is injected into the
scheduler once per session (worker push, cached in w->resvals/
w->resInUse); a single session then evaluates many jobs against many
candidate hosts off that one cached snapshot without re-querying
between decisions. Made the reason explicit: a live query per
candidate-host check, at the volume a session performs, would be far
too costly. This is a documentation clarification only -- no
behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
A job with no rusage[mem=...] request (reserved=0) was being SKIPPED
entirely during reservation-vs-usage reconciliation, on the mistaken
assumption that "no reservation" meant "nothing needs charging". It
doesn't: this codebase's admission check has no live OS read at
decision time (sched_check_host_satisfy_(), resource.c) -- it compares
the worker's last-reported "mem" value against this scheduler's OWN
tracked resInUse tally, built entirely from summed xce_res_account_t
entries. Skipping an unreserved job left its real, live memory
consumption permanently invisible to that tally -- an actual
over-subscription risk (the scheduler could admit more onto a host
than it can support, believing memory was free that a live process was
genuinely using), not merely a display gap.
Fix: charged = max(reserved, actual) applies to EVERY job, reserved or
not. ensure_host_account_by_resname() (resource.c) now creates a fresh
reserved=0 account entry (resolving residx via key_find() against
core->resmgr, same pattern sched_update_res_accounts() already uses
for a fresh account) instead of skipping when a job has no existing
entry for a resource. Returns NULL (a genuine no-op) only when the
resource isn't registered in the cluster's config at all.
Rewrote resource-accounting.md to correct the mechanism it documented:
"available" is NOT a live OS read (mem vs the similarly-named but
static maxmem was worth naming explicitly too) -- it's worker's
last-reported mem minus this scheduler's own resInUse tally, and that
tally must include every job's real usage regardless of reservation.
Also documented a known, pre-existing property surfaced while
correcting this: since mem's own worker report already reflects real
consumption and the scheduler's charge is a separate layer that
doesn't get cleared on a fresh report, tracking can become
conservative over time (under-utilization risk, never over-
subscription) -- the same direction slots already accepts, and a
separate, larger question than this fix's scope.
Replaced test_reconcile_job_resource_usage_skips_when_no_mem_request
(asserted the now-wrong skip behavior) with two tests: one proving an
unreserved job's real usage is created and charged, one proving a
genuine no-op when "mem" isn't a registered resource at all. Full
scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.
Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.
The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.
New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).
Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.
Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.
Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).
Full scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
All 11 leaf tasks (1.1-4.5) are checked done and the feature was
closed out by its own commits (5948e079, b64682dc); spec.json still
read "tasks-generated" -- same phase/reality lag class already fixed
for other specs this week.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
test_live_config_push_against_standby_returns_standby_rejection_not_redirect
locked in the OLD controller behavior (a standby flatly rejects
OP_upgradeConfig with no redirect fields). That gap is now closed:
myreq_handler's gate-rejection path calls fill_redirect_reply() on
every -XCE_ERR_STANDBY rejection (bug-1700), and handle_ctrlhello() now
actually populates primary_host/primary_port from the static --peers
list (bug-1701) -- a standby pushing config now redirects to the
primary and the push succeeds. Renamed to
test_live_config_push_against_standby_redirects_and_succeeds and
rewritten to assert the new behavior.
_assert_primary_committed() changed from a single-shot log read to a
poll loop (up to 5s) for the "config activated from committed
marker..." line -- accounts for the documented async gap between the
majority-ack-gated client reply and the primary's own local WAL-apply-
pump activation, a real race the old single-shot read could hit.
meson-build.sh: wire src/tools and each build profile's controller
subdir into the isolated repo-mirror the E2E tests run against, and
export XCEADMIN_BIN explicitly, so these live tests actually find real
binaries instead of always skipping.
Verified: xceadmin suite 288/288 (incl. both tests in this file);
controller suite 1220/1222 (only the pre-existing bug-1561 flake).
Updated admin-config-tool design.md and roadmap.md to mark the
dispatch-gate and redirect-wiring gaps RESOLVED, matching this test's
new assertions and the live evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Requirements, design, research, and tasks all generated and approved
(ready_for_implementation: true). Implementation not yet started (0/8
tasks) -- saving the completed spec authoring; the 8-task
implementation is separate future work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds the previously-uncommitted spec.json/requirements.md/design.md/research.md
for scheduler-snapshot-job-allocation (feature now fully implemented across
tasks 1.1-4.5) and the in-progress xce-test-cli-ux spec's requirements.md/
spec.json. Updates brief.md, a security.md note on an unrelated deferred RBAC
persistence gap found during this feature's cross-component survey, and syncs
OpenWolf's anatomy/memory/token-ledger/daemon-log bookkeeping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.5: scheduler dimension-4 field-completeness audit run against the
finished implementation -- clean, no undocumented gap. Corrects the one
remaining stale premise in snapshot.c's "internal_state contents" comment
(claimed job->allocs/accounts were still unrestored; they are now, per
task 2.2) while preserving its still-valid sharedInUse/resInUse rationale.
All tasks (1.1-4.5) for scheduler-snapshot-job-allocation are complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.4: the "Scope decisions" header comment previously stated a
job's allocs/accounts are "not captured" -- update it to describe the
now-implemented dup_job_alloc()/make_job_account() restoration path,
and add a note documenting job->rusages as a known, still-uncaptured
gap requiring a future capnp schema change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.3: independent full scheduler+lib suite rerun confirms zero new
failures beyond pre-existing unrelated bug-1560 and bug-1723; scope
confined to allocation/schedobj paths with zero diff to snapshot.c or
the reconciliation test files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.2: extend test_e2e_original_motivation_long_replay with PREEXIST_ID
(job 70013, dedicated worker worker-e2e-pre) -- a job submitted,
live-dispatched, and confirmed RUN entirely before sched_record_start(),
so its dispatch decision can never appear in the replay stream. Proves
its "used=9.00" resource contribution is correct starting from the very
first replayed round, recoverable only via snapshot-load allocation
restoration (tasks 2.1/2.2), not replay-based reconciliation -- the
complementary case to the file's existing ANCHOR_ID scenario.
Completes and fixes a pre-existing uncommitted WIP addendum to this file:
removed a redundant enqueue_queue_update() call that was masking a
genuine, unrelated latent bug in core_upsert_queue() (rtti makeString()
pointer freed via plain free(), same class as bug-1503/bug-1600) --
noted in tasks.md's Implementation Notes as a follow-up, not fixed here
(test-only task boundary).
Verified via fault injection: gating snapshot_load.c's allocation
restore reproduces the exact pre-fix failure at this test's new
assertion, then cleanly reverts.
Requirements: 1.1, 1.2, 1.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BSiC2cPVH2VSBdtFq4P4ju
Task 4.1: new test proving a snapshot-restored PENDING job (valid
resreq) reaches the real, live sched_allocator() thread's
sched_make_allocation() entry point without crashing -- the exact
bug-1569-class failure task 1.2 closed for the snapshot-load path.
New file (not an extension of test_snapshot_save_load.c) since driving
the real allocator needs scheduler.c's full dependency closure, too
heavy for that file's deliberately lean core-only TU. Follows the
already-established pattern from test_scheduler_connect.c /
test_scheduler_backpressure.c / test_scheduler_dispatch_commit.c: link
the real scheduler.c, wrap xcecli_start/xcecli_join, start a real
sched_init(), feed jobs directly into core.jbuf to drive the live
allocator thread. Wired into meson.build as a new executable/test.
Verified with two fault-injection experiments (reverted after):
disabling task 1.2's schedobj construction call makes this test fail
cleanly via its own assertion; going further and removing that
assertion reproduces a genuine ASAN-caught SEGV in the real allocator
path -- concrete proof this test would have caught bug-1569's failure
shape. Both polling loops (jbuf drain, decision wait) are bounded
(2s timeout, 5ms poll) -- no hang risk; confirmed non-flaky across 10
consecutive runs.
Requirements: 2.2, 2.4
Task 3.2: adds test_snapshot_load_leaves_running_job_without_allocation_at_zero_resinuse,
covering the specific Req 3.5 scenario the existing PENDING-job test didn't
-- a RUNNING job whose allocation was never captured (simulating a
pre-feature-saved snapshot). With a real worker/resource present (making
the assertion non-vacuous), confirms: load succeeds with no error,
job->schedobj is still valid (task 1.2, independent of allocation data),
job->allocs/accounts stay NULL, and resInUse correctly reads 0.0 via the
real accounting chain -- a clean degrade, no special-casing, matching
today's pre-feature behavior exactly.
Test-only task; no production code changed. Per this spec's own
"pre-release product" scope note, no archived legacy snapshot file is
needed as a fixture -- a freshly-built wire job with allocs unset
exercises the identical code path.
Requirements: 3.5
Task 3.1: comprehensive integration test proving tasks 1.1/1.2/2.1/2.2
work together end to end -- a RUNNING job (multi-host, multi-resource,
plus a genuine shared-resource-pool component) and a PENDING job saved
together and loaded into a second, independent core. Asserts:
- Both jobs get a valid, non-NULL schedobj in the restored core.
- The running job's allocation/accounting content-match the source,
including the shared-pool component.
- Worker resInUse and core sharedInUse, computed via the real
sched_charge_account() chain (white-box included from resource.c,
following the established test_resource_accounting.c/test.c
link-only-stub convention), match exactly between source and
restored cores -- the actual proof of Req 1.2/1.3, not just that
job->accounts holds the right values.
- Destroying the source core's job list afterward does not corrupt or
crash the restored core (Req 1.4's independence guarantee exercised
via real teardown, not just pointer-distinctness checks).
Test-only task; no production code changed. Independently verified via
a live RED/GREEN toggle against task 2.2's load-side restoration block.
Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.3, 2.4
Task 2.2: xce_sched_core_load_snapshot() now restores job->allocs via
dup_job_alloc() (never aliasing the wire job's own copy, since it
belongs to a const, potentially-shared snapshot struct) and rebuilds
job->accounts via the existing make_job_account(), positioned right
after task 1.2's schedobj construction and before sched_add_job().
Independent of resreq parse success -- not nested inside that guard.
Absent allocation data is the normal case: no WARN, no error.
New tests prove a full cross-core round trip (save core_a, load into
independent core_b): restored allocs/accounts content-match the
source, with three-way pointer independence (restored vs source-core
vs saved-wire) proving the deep-copy guarantee end to end. A companion
test confirms a job with no allocation loads cleanly.
Note: this task's implementation was completed directly after two
implementer subagent dispatches hit transient API interruptions; one
auto-resumed and reported a false READY_FOR_REVIEW (claiming 9/9
passing tests while the actual production code was never written,
only the test file). Caught by fresh-evidence verification before
trusting the report -- confirmed via an independent build+test run
showing a genuine failure, then fixed and re-verified by an
independent reviewer who reproduced RED-to-GREEN itself.
Requirements: 1.1, 1.2, 1.3, 1.4, 3.5
Task 2.1: build_wire_job() now deep-copies job->allocs into the wire
job's already-existing allocs field via dup_job_alloc() (task 1.1) when
the live job has an allocation, leaving it unset otherwise -- matching
how every other optional field in this function is handled. Never
aliases job->allocs.
New tests prove: a running job with a multi-host allocation saves with
matching content (host names, resource values) and a pointer distinct
from the live job's own allocation; a job with no allocation saves with
the wire field left NULL, no error.
Requirements: 1.1
Task 1.2: snapshot_load.c gains a local, static field-for-field port of
core_events.c's core_build_schedobj()/core_build_primary_schedobj()/
core_build_compound_schedobj()/core_build_alter_schedobj() (the bug-1569
fix for the JOBS_PUSH path). Every job restored from a snapshot whose
resreq parsed successfully now gets a valid, non-NULL job->schedobj,
closing the same NULL-deref crash class for the snapshot-load path.
Guarded on job->resreq != NULL so the pre-existing, unrelated
resreq-parse-failure path is untouched (still leaves schedobj NULL,
same as today -- not fixed here, per design.md's documented boundary).
New tests prove non-NULL schedobj with correct tag/union member for
primary, compound, and alter resource requests, plus a regression test
confirming the parse-failure path's behavior is unchanged.
Requirements: 2.1, 2.2, 2.3, 2.4
Task 1.1: new deep-copy for xce_job_alloc_t composed from existing
make_alloc/make_host_alloc/make_job_alloc, explicitly preserving each
allocation's original from/duration rather than accepting make_alloc's
own time(NULL) stamp. TDD: 6 new unit tests (single-host, multi-host,
multi-resource-per-host, shared-resource-pool, independence-after-free,
from/duration fidelity), all passing under ASAN.
Also logs bug-1723 (pre-existing, unrelated leak in two bug-1722 test
fixtures, confirmed via git-stash A/B rebuild to be byte-for-byte
identical before and after this change) -- documentation only, not fixed
here.
Requirements: 1.1, 1.4
Prompted by rediscovering the same bug class in two places this
session: the controller's WAL already has this documented for one
mechanism (persistence.md's APPLY_CLEAN/bug-1678 section, xce-audit-
invariants controller dimension 7) but the scheduler's separate
snapshot save/load mechanism had the identical gap (job->allocs, this
session's scheduler-snapshot-job-allocation brief) with no equivalent
tracking. Closes that gap and makes the invariant self-reinforcing
going forward rather than something that has to be independently
rediscovered per persistence mechanism:
- xce-audit-invariants: new scheduler dimension 4 ("Field-complete
snapshot save/load parity"), sibling of controller dimension 7,
scoped to snapshot.c/snapshot_wire.c/snapshot_load.c.
- persistence.md: new "Snapshot Save/Load Field-Completeness
(Scheduler)" section cross-referencing the WAL-side bug-1678 writeup
and the new scheduler dimension.
- CLAUDE.md: new top-level Key Rules bullet stating the invariant
plainly, plus adding persistence.md to the steering-file pointer
table (it was missing).
- kiro-spec-design's design-review-gate.md (the mandatory pre-write
gate) and kiro-validate-design's design-review.md (the optional deep
review): new conditional review sections -- applies only when a
design touches a save/restore mechanism, requires enumerating every
persisted field's capture status rather than trusting an aggregate
"state looks right" check. Catches this class of gap at design time,
before implementation, rather than only via a later audit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
/kiro-discovery investigation of removing src/frontend/Makefile in favor
of Meson as the sole build system. Records the quick investigation's
findings (5 boundary seams, current-state evidence that build-cli/
build-server are already dead from /xce-build's perspective and
build-server is already broken today) and the user's confirmed decision
to fully remove the Makefile rather than keep a compat wrapper.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
CLAUDE.md: replace the stale @.wolf/OPENWOLF.md import with a pointer to
the openwolf skill / rules file, matching how the project's OpenWolf
setup actually resolves the protocol now.
api-standards.md: add checklist item 6 for new primary-only opcodes --
confirm ha_dispatch_gate's -XCE_ERR_STANDBY rejection actually populates
a redirect via fill_redirect_reply(), not just that the client has a
generic redirect-follower. Directly informed by bug-1700, where the
identical gap was independently rediscovered for OPCode_schedAdmin after
already being documented once for OPCode_upgradeConfig.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Pure verification, zero code/test changes. Confirmed: the 9 pre-existing
test_sim_reconcile_unaccounted.c tests pass unchanged; interactive/batch
parity (test_repl_equivalence.c, untouched by this spec) passes; full
scheduler suite shows the identical 53 Ok / 1 Fail baseline (sole failure
the pre-existing, unrelated bug-1560), with the spec's 6 net-new tests
(tasks 1.1/1.2/2.1/3.1/3.2) as the only addition to that baseline.
Marks all tasks in the scheduler-sim-eager-reconciliation spec complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_sim_verify_no_false_mismatch_across_round_separated_dispatch:
a job's DISPATCH decision at round 1, RUN-state confirmation event at
round 4, --sim-verify enabled throughout, asserting zero mismatches.
Fixture correctly accounts for a mechanically-verified system behavior:
sim mode's live picker (sim_picker_thread_fn/builtin_fcfs) never emits
on_dispatch() -- it unconditionally re-PENDs every candidate each round,
including a job reconciliation already captured, until the job's
recorded JOB_STATE_CHANGE event removes it from pendJobs. The fixture
includes a matching PEND(jobid) entry at each intervening round to match
this reality, not an assumption.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap:
a worker at exact capacity (jobs A=8.0 + C=2.0 against 10.0 capacity), with
each job's RUN-state confirmation landing several rounds after its DISPATCH
decision (round 1 dispatch, round 4/5 confirmation, deliberately different
gaps). Drives the real sched_check_host_satisfy() admission check at three
checkpoints, proving Requirement 1 (1.1-1.4) at the outcome level -- a
capacity check, not just a resInUse display value -- per
scheduler-driver-equivalence.md's explicit contention-fixture discipline.
Round 1 review caught three new xce_sched_sim_driver_t fixtures declared
stack-local, amplifying an already-documented (bug-1598, bug-1632)
stack-use-after-return class from the never-joined picker thread
sched_sim_driver_run() spawns. Fixed by declaring them static (safe: each
is written by exactly one call site, and sched_sim_driver_start() memsets
the struct first) -- independently confirmed via 3 full-suite runs with
zero ASAN hits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds 3 tests proving guarantees that must hold under the widened
capture-eligibility gate: an already-captured job is never re-searched
or re-captured regardless of state (test_reconcile_still_skips_already_
captured_job_regardless_of_state), a PEND job with no match this round
stays uncaptured and remains eligible next round (test_reconcile_
no_match_leaves_pend_job_uncaptured_and_retries), and several
simultaneously outstanding jobs are captured independently with zero
cross-job interference (test_reconcile_multiple_jobs_captured_
independently_across_different_rounds).
Required making the shared add_job() fixture helper tolerant of -EEXIST
on a repeated queue registration (needed for the multi-job test, the
first in this file to call add_job() twice against the same core) --
traced to kvtab_add()'s narrowly-scoped -EEXIST return, provably a
no-op for all 11 pre-existing single-add_job()-call tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_captured_pend_job_excluded_from_resinuse_until_run: drives a
job captured while still PEND through sched_account_jobs() and confirms
its resource-in-use contribution is zero; transitions the same job to
RUN and confirms sched_account_jobs() now includes its exact captured
allocation, with no second reconciliation call in between -- proving the
nonzero value comes from the earlier capture, not a fresh one.
Required promoting the file's sched_account_jobs() from a no-op stub to
a real reimplementation (verbatim body, matching the exact precedent
already established in test_sim_contention_parity.c and
test_sim_round_boundary_reconciliation.c) plus a resource.c white-box
include -- no production code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Widens sim_reconcile_unaccounted_jobs()'s trigger gate in sim_loop.c: removes
!JOB_IS_RUN(job->state) so a job's recorded DISPATCH decision is captured as
soon as the replay reaches the round it occurred in, regardless of the job's
current state -- closing bug-1611/bug-1613's residual (a job whose RUN
confirmation lands in a later round than its dispatch decision, the realistic
case under any real dispatch-to-confirmation latency, was never reconciled).
sched_account_jobs()'s own unmodified JOB_IS_RUN charging gate continues to
defer actual resource-in-use accounting until the job genuinely transitions
to RUN -- no other line in the match/capture/report path changes.
Replaces test_reconcile_skips_non_run_state_job (which asserted the bug as
correct behavior) with test_reconcile_captures_non_run_state_job_when_dispatch_recorded,
proving the opposite for a PEND-state job. Adds the xce-test-rc-file-style
spec docs (requirements/design/research/tasks) for this feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
No product/spec-test code changed. Ran the real xce-test-ha soak and
xce-test-resources verification drivers with no rc file present anywhere
in the environment they run under -- both pass their own documented
success criteria unchanged (zero job loss across rolling failover; zero
resource-value mismatches across controller/scheduler/apiserver), proving
Requirement 1.3's "no rc file -> unchanged behavior" guarantee holds for
real, pre-existing automation that has no awareness this feature exists.
Marks all tasks in the xce-test-rc-file spec complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
