Persistent log of the resource-design discussion: the user's proposals
(generic supply map incl. provisioned supply; a resource provider proxy
between the controller and upstream providers; datasets in the map with
staging as data provisioning) with analysis, the user's 2026-10-11 input
(duration/decay must-have but future; RBAC in the apiserver, discuss later;
OCI containers via application profiles; CSI compatibility preferred;
GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items
and new questions N1-N4, C3. resource-model.md, the provisioner research,
roadmap and CLAUDE.md point at it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
resource-model.md: one resource type descriptor and provider contract over
three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler
only selects from its cached snapshot, providers run after dispatch; batch
stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims
as WAL entities with the resource-provisioner discipline; LSF/K8s mapping;
LLM training/serving needs; phases; nine pending user decisions.
New briefs: custom-resource-reporting, resource-instances,
resource-claims, gang-groups. Roadmap: long-term LLM training + serving
platform direction (features from LSF and Kubernetes), the four new specs,
updates to ten existing specs, discovery note. resource-distribution-mapping
becomes phase 1 (shared pools + descriptor fields).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scaler is multi-threaded (monitor and demand threads); invoke() forked
and, in the child, logged, snprintf'd and setenv()'d before execl(), which
can deadlock on a lock another thread held at fork time and block the
caller in waitpid() forever.
run_script() now posix_spawn()s `sh <script> -f <in.json>`: stdout and
stderr go to the output file through file actions, the provider variables
(PRO_CONF_DIR, PRO_DATA_DIR, PRO_LOG_DIR, PROVIDER_NAME) are passed as the
child's envp (provider_env()), the child starts with an empty signal mask
and default dispositions, and waitpid() retries on EINTR.
The scaler unit suite (scaler/testsuite) was never built; it is now the
meson test "autoscaler test", linked with -Wl,--wrap=fork so it counts
fork() calls during a provider call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Design constraint (tech.md "Process Creation"): the controller, scheduler,
xcecd and anything that starts libxce threads never call fork, vfork,
forkpty, daemon, system, popen or uv_spawn; children start with posix_spawn()
or a systemd unit. A process proven single-threaded may fork, marked
/* xce-fork-ok: <why> */. Tests, harness, autoscaler (not finalized; bug-1969)
and the Go frontend are out of scope.
- scripts/check-no-fork.py scans src/{controller,scheduler,lib,worker}
(comment/string/#if 0 aware); runs as meson test "no-fork rule" in the
lib/controller/scheduler/worker suites and in the pre-commit hook.
- worker: drop xcecd's unused fork-based worker start paths
(process_manager_start_process, process_manager_start_worker; xcecd
starts xcecd_worker over D-Bus) and a dead #if 0 daemonize fork.
- xcecd_worker's fork/uv_spawn/system calls are marked: its call graph
from main(), callbacks included, reaches no pthread_create or libuv
thread-pool call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
deployment.md: auth.yaml now holds bcrypt hashes of the (still public)
dev credentials with startup warnings; JWT secret from XCE_JWT_SECRET or a
random per-process secret; logout revocation is per process;
security.trusted_proxies for reverse proxies.
security.md steering: JWT parsing, revocation, secret resolution,
trusted proxies, middleware abort rule, bcrypt and role-change rule.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce-node-manager: a "Cross-Spec Interactions" section (pending a
requirements revision; approved criteria unchanged) with six points: a
supervision/probe module shared with service-supervision, node-manager
components vs service jobs, a lease-based provisioner role, role
declaration and worker stop on decommission for provisioned hosts, a
host-loss grace period longer than worker restart plus rejoin, and a future
service proxy under the reverse-proxy role. design.md Open Questions points
to it.
The service-supervision, host-failure-job-recovery, elastic-hosts,
resource-provisioner and service-endpoint briefs cross-reference the
matching points; the roadmap lists the xce-node-manager revision.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New briefs: provisioning-demand (the scheduler knows a provisionable
catalog, tries host templates as virtual hosts and license headroom after
real resources fail, finalizes demand and idle supply at session end;
simulator support is a hard requirement), resource-provisioner (new
daemon, several instances owning providers via controller leases,
self-describing providers, automatic policy, durable request/instance
state; src/autoscaler is prior art only), elastic-hosts (host templates,
expected hosts, drain and decommission), elastic-shared-resources
(borrowed license tokens change the shared total).
Roadmap: the new specs, plus updates to resource-distribution-mapping
(shared pools) and peer-authentication-hardening (provisioner role).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New briefs: host-failure-job-recovery (jobs on a lost host are requeued or
end EXIT after a grace period; stale copies fenced on rejoin; manual
requeue), service-supervision (profile kind service: restart policy and
liveness/readiness probes run by each replica's own worker, in-place
restart), service-replica-set (controller-held service object owning
independent replica jobs; event-driven replacement, scaling and rolling
update; no head agent, no promotion), service-endpoint (per-replica ports,
endpoints of ready replicas), job-resize (backlog, elastic jobs).
application-profile brief gains the service section; roadmap lists the new
specs, the existing-spec updates and the backlog item.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
job-config-merge: spec initialized from its discovery brief.
project-config: requirements generated (9 requirements): nested projects
named by path, membership-based submit check with no inheritance and
overlapping membership, group-default then cluster-default project, open
mode until projects are defined, pending jobs held with a reason after
project changes, roll-up quotas, project visibility, a project scope for
job permissions covering the subtree and a project:view permission,
compatibility.
Roadmap: project-delegated-membership backlog item (runtime membership
edits by project managers, deferred).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New steering job-config-layering.md: queue = scheduling policy, application
profile = what the application needs and how it runs (resreq, task range,
execution settings, pre/post-exec, limits), project = resource ownership.
Only two interactions: the job overrides a profile attribute marked
default (fixed: not at all; select: only ANDed terms), and queue ceilings
or project quotas check effective values (reject if never satisfiable,
pend if full now). Replaces the simplified job > app > queue rules.
Queue pre_exec/post_exec are deprecated with queue res_req: the profile's
pre/post-exec fills the worker's existing outer layer, the job's own stays
the inner layer, allowed or forbidden by the profile.
job-config-merge and application-profile briefs and the roadmap follow;
CLAUDE.md lists the new steering file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New briefs: job-config-merge (one live merge of job, application profile,
queue policy and project into effective values; simplified rules: job
overrides profile, select ANDed, queue hard/task limits reject; queue is
policy only, res_req deprecated), project-config (project quotas v1) and
project-guarantee (backlog).
application-profile: the profile carries the job's resreq (incl. span) and
task limits; "frozen at submit" replaced by the live merge.
parjob-placement: span constraints live in the fixed-N planner and the
range search bisects over the planner's allowed counts; LSF 10.1 block,
ptile, stripe and HOSTLIMIT_PER_JOB findings recorded.
Roadmap: new specs, existing-spec updates, lsf-config-migration backlog.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
src/docs/xce.md gains a 'Job arrays' section: name[1-N] and name[1],
numbering from 1, the API-server limit (default 1000, at most 32767),
<array-id>[k] and <array-id>[a-b] selection (with the peek/restart
caveat), XCE_JOBINDEX, the harness jobsub -a N / $jobid[k], and every
LSF array feature this release does not support. The superseded
statements in job-array-single-wal-record and xce-test-e2e-primitives
are annotated (history kept), the controller-job-cleanup and
job-dependency briefs describe 1-based numbering, and the roadmap gains
an entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Feature validation GO (ASAN scheduler suite 96/96, smoke replay clean on
the allocation path). Roadmap entry marked implemented with follow-ups.
bug-1897 (found by the validation smoke run, not caused by this feature):
xce_scheduler --sim segfaults in limit_job_election on a job whose
snapshot resreq failed to parse (pre-bug-1888-fix replays).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New specs from /kiro-discovery (2026-09-29), based on the LSF 10.1 Admin
Guide section "Job dependency and job priority". Dependency only; job priority
is deferred to a later discovery. Both are added to roadmap.md.
- job-dependency: -w expressions by job ID (done, ended, exit(id[,op code]),
started, post_done, post_err, bare id; && || ! and parentheses). One libxce
grammar, used from Go through cgo. The apiserver validates early (syntax,
parent exists, job:view). The controller re-validates, stores the
expression, and keeps an in-memory reverse index rebuilt on replay. The
scheduler evaluates in sched_job_is_ready(); unsatisfiable dependencies stay
PEND. Also closes the scheduler exit-code field-completeness gap. Adds
xcecli -w / modify -w -wn and harness jobsub -w / jobmod.
- job-dependency-insight: bjdepinfo-style query, dependency info in job
details, and optional orphan termination (-ti or a cluster grace period).
- controller-job-cleanup brief: job-dependency will replace the
job_has_dependents() stub.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New spec from /kiro-discovery (2026-09-29), added to roadmap.md:
- Begin time: the scheduler won't start a job before it (gate in
sched_job_is_ready() using the core clock). New pend reason
BEGIN_TIME_NOT_REACHED.
- End time: the worker enforces it for running jobs (SIGTERM, grace, SIGKILL)
and records a persisted END_TIME_REACHED exit reason. The primary is not
involved. A pending job whose end time can't be met stays PEND with
END_TIME_UNMEETABLE until the user kills or modifies it.
- Runtime estimate (-We): a scheduling hint only. It sets the rule
now + (estimate, else run limit, else 0) > end; an estimate larger than the
run limit is rejected.
- Times and estimate can be changed only while the job is pending, through a
new `xce jobs modify` in xcecli and a new `jobmod` in the harness. Harness
jobsub gains all four options.
- Fixes the silent drop of xcecli -b/-t/-W and REST run_time_estimation,
which the apiserver's modelJobToSubmit never passes to the controller.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New specs from /kiro-discovery (2026-09-28/29), added to roadmap.md:
- job-status-detail-display: show pending reasons and reserved-vs-used
resources in REST and xcecli. Requirements generated (7 requirements).
- scheduler-alter-pend-reasons: keep a pending reason for every alternative
resreq (per-alternative JobPendReason) and fix the obj->reason leak.
- controller-job-cleanup: primary-only sweep that removes finished jobs past
a configurable retention, via a replicated WAL removal fact.
- application-profile: admin-defined application profiles (execution mode,
launch type, env, resource defaults) chosen per job, with app:use/app:view
RBAC. Queues and profiles stay orthogonal.
- job-sandbox-execution / job-sandbox-multinode: run jobs in a systemd
DynamicUser sandbox behind a pluggable execution backend.
- rbac-persistence: make RBAC changes made through the API survive an
apiserver restart (closes the security.md gap).
roadmap.md also records the scheduler-sim-diagnostics info job update and a
security item: remote-task units appear to run as root (found by code
reading).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The dimension-1 audit (previous commit) reported bug-1611 as only
"partially fixed" -- but that assessment worked from bug-1611's own
stale narrative text without checking scheduler-sim-eager-reconciliation
(a separate spec, completed 2026-09-11, predating this session's audit).
That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger
from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in
the same round it is recorded, never deferred to a later confirmation
event -- closing bug-1611's original motivating scenario for good.
The "residual gap" my own first correction pass invented (an
already-RUN-at-snapshot-load job) is also not real: such a job's
allocation comes from the snapshot directly (confirmed field-complete
by this session's dimension-4 audit), so it's never a reconciliation
candidate in the first place.
Corrected bug-1611's buglog entry to FIXED and updated
scheduler-driver-equivalence.md's stale "Known violations" section
(still described both bug-1611 and bug-1613 as open) to reflect current
reality, with a pointer to re-verify rather than re-trust either framing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Validation task 4.3 found a real, deterministic conflict with the
original requirement 2.2 wording, not a flake: /xce-build controller
alone's default asan profile transitively rebuilds harness/xce_test,
because Meson's test-runner rebuilds every registered test's build-
order dependencies project-wide before filtering by --suite, and
harness_unit_test (task 2.1) depends on it. Confirmed via build-log
step-count analysis (explicit compile: 209 steps, clean; subsequent
meson test: 844 steps, resurrects it) and independently reproduced
twice, with an exact step-count match both times.
User decision (2026-09-14): accept as a documented Meson limitation
rather than auditing every component's test-target list (controller's
own suite already relies on this same mechanism for 4 of its own test
binaries) or weakening harness_unit_test's build-order guarantee
(verified load-bearing in task 4.2). Requirement 2.2 narrowed to what
the build system directly controls: its explicit compile-target list
never names xce_test. requirements.md/design.md/tasks.md updated to
match and cross-reference each other; logged as bug-1806 (accepted,
not fixed) for the record.
Also commits the xce-harness-component spec's remaining docs
(requirements.md, brief.md, research.md, spec.json) and the roadmap.md
promotion entry from discovery, none of which had been committed yet.
Task 4.3 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Add harness to structure.md's top-level src/ directory tree (after
controller/) and extend the flat-root Exceptions bullet to cover it
alongside controller. Verified against the real src/harness/ layout
(14 flat .c files, no nested src/, own include/ and testsuite/).
This completes Major Task 3 (Integration) -- all downstream skills and
steering docs now correctly describe the harness component.
Task 3.7 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Add harness to root CLAUDE.md's /xce-build component list and
Subdirectory Guidelines (pointing at src/harness/CLAUDE.md). Remove
build-deployment.md's false "controller always builds xce_test" claim
and add harness to its /xce-build command table, Meson Structure tree,
dependency-flow line, and worked compile-target example.
Review round 1 caught a third surviving instance of the same stale
claim in the bug-1565 illustrative paragraph, contradicting the
corrected text two paragraphs above it -- fixed to reference the
binaries controller actually produces (xce_controller,
controller_unit_test) instead of xce_test.
Task 3.6 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.
Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Expose the AppImage as a flake output via the nix-appimage input's
mkAppImage (passthru on the xce package) instead of requiring an
external 'nix bundle --bundler github:ralismark/nix-appimage' call.
Update README and build-deployment steering to match.
Resource information for candidate hosts is injected into the
scheduler once per session (worker push, cached in w->resvals/
w->resInUse); a single session then evaluates many jobs against many
candidate hosts off that one cached snapshot without re-querying
between decisions. Made the reason explicit: a live query per
candidate-host check, at the volume a session performs, would be far
too costly. This is a documentation clarification only -- no
behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
A job with no rusage[mem=...] request (reserved=0) was being SKIPPED
entirely during reservation-vs-usage reconciliation, on the mistaken
assumption that "no reservation" meant "nothing needs charging". It
doesn't: this codebase's admission check has no live OS read at
decision time (sched_check_host_satisfy_(), resource.c) -- it compares
the worker's last-reported "mem" value against this scheduler's OWN
tracked resInUse tally, built entirely from summed xce_res_account_t
entries. Skipping an unreserved job left its real, live memory
consumption permanently invisible to that tally -- an actual
over-subscription risk (the scheduler could admit more onto a host
than it can support, believing memory was free that a live process was
genuinely using), not merely a display gap.
Fix: charged = max(reserved, actual) applies to EVERY job, reserved or
not. ensure_host_account_by_resname() (resource.c) now creates a fresh
reserved=0 account entry (resolving residx via key_find() against
core->resmgr, same pattern sched_update_res_accounts() already uses
for a fresh account) instead of skipping when a job has no existing
entry for a resource. Returns NULL (a genuine no-op) only when the
resource isn't registered in the cluster's config at all.
Rewrote resource-accounting.md to correct the mechanism it documented:
"available" is NOT a live OS read (mem vs the similarly-named but
static maxmem was worth naming explicitly too) -- it's worker's
last-reported mem minus this scheduler's own resInUse tally, and that
tally must include every job's real usage regardless of reservation.
Also documented a known, pre-existing property surfaced while
correcting this: since mem's own worker report already reflects real
consumption and the scheduler's charge is a separate layer that
doesn't get cleared on a fresh report, tracking can become
conservative over time (under-utilization risk, never over-
subscription) -- the same direction slots already accepts, and a
separate, larger question than this fix's scope.
Replaced test_reconcile_job_resource_usage_skips_when_no_mem_request
(asserted the now-wrong skip behavior) with two tests: one proving an
unreserved job's real usage is created and charged, one proving a
genuine no-op when "mem" isn't a registered resource at all. Full
scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.
Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.
The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.
New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).
Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.
Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.
Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).
Full scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
test_live_config_push_against_standby_returns_standby_rejection_not_redirect
locked in the OLD controller behavior (a standby flatly rejects
OP_upgradeConfig with no redirect fields). That gap is now closed:
myreq_handler's gate-rejection path calls fill_redirect_reply() on
every -XCE_ERR_STANDBY rejection (bug-1700), and handle_ctrlhello() now
actually populates primary_host/primary_port from the static --peers
list (bug-1701) -- a standby pushing config now redirects to the
primary and the push succeeds. Renamed to
test_live_config_push_against_standby_redirects_and_succeeds and
rewritten to assert the new behavior.
_assert_primary_committed() changed from a single-shot log read to a
poll loop (up to 5s) for the "config activated from committed
marker..." line -- accounts for the documented async gap between the
majority-ack-gated client reply and the primary's own local WAL-apply-
pump activation, a real race the old single-shot read could hit.
meson-build.sh: wire src/tools and each build profile's controller
subdir into the isolated repo-mirror the E2E tests run against, and
export XCEADMIN_BIN explicitly, so these live tests actually find real
binaries instead of always skipping.
Verified: xceadmin suite 288/288 (incl. both tests in this file);
controller suite 1220/1222 (only the pre-existing bug-1561 flake).
Updated admin-config-tool design.md and roadmap.md to mark the
dispatch-gate and redirect-wiring gaps RESOLVED, matching this test's
new assertions and the live evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds the previously-uncommitted spec.json/requirements.md/design.md/research.md
for scheduler-snapshot-job-allocation (feature now fully implemented across
tasks 1.1-4.5) and the in-progress xce-test-cli-ux spec's requirements.md/
spec.json. Updates brief.md, a security.md note on an unrelated deferred RBAC
persistence gap found during this feature's cross-component survey, and syncs
OpenWolf's anatomy/memory/token-ledger/daemon-log bookkeeping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Prompted by rediscovering the same bug class in two places this
session: the controller's WAL already has this documented for one
mechanism (persistence.md's APPLY_CLEAN/bug-1678 section, xce-audit-
invariants controller dimension 7) but the scheduler's separate
snapshot save/load mechanism had the identical gap (job->allocs, this
session's scheduler-snapshot-job-allocation brief) with no equivalent
tracking. Closes that gap and makes the invariant self-reinforcing
going forward rather than something that has to be independently
rediscovered per persistence mechanism:
- xce-audit-invariants: new scheduler dimension 4 ("Field-complete
snapshot save/load parity"), sibling of controller dimension 7,
scoped to snapshot.c/snapshot_wire.c/snapshot_load.c.
- persistence.md: new "Snapshot Save/Load Field-Completeness
(Scheduler)" section cross-referencing the WAL-side bug-1678 writeup
and the new scheduler dimension.
- CLAUDE.md: new top-level Key Rules bullet stating the invariant
plainly, plus adding persistence.md to the steering-file pointer
table (it was missing).
- kiro-spec-design's design-review-gate.md (the mandatory pre-write
gate) and kiro-validate-design's design-review.md (the optional deep
review): new conditional review sections -- applies only when a
design touches a save/restore mechanism, requires enumerating every
persisted field's capture status rather than trusting an aggregate
"state looks right" check. Catches this class of gap at design time,
before implementation, rather than only via a later audit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
/kiro-discovery investigation of removing src/frontend/Makefile in favor
of Meson as the sole build system. Records the quick investigation's
findings (5 boundary seams, current-state evidence that build-cli/
build-server are already dead from /xce-build's perspective and
build-server is already broken today) and the user's confirmed decision
to fully remove the Makefile rather than keep a compat wrapper.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
CLAUDE.md: replace the stale @.wolf/OPENWOLF.md import with a pointer to
the openwolf skill / rules file, matching how the project's OpenWolf
setup actually resolves the protocol now.
api-standards.md: add checklist item 6 for new primary-only opcodes --
confirm ha_dispatch_gate's -XCE_ERR_STANDBY rejection actually populates
a redirect via fill_redirect_reply(), not just that the client has a
generic redirect-follower. Directly informed by bug-1700, where the
identical gap was independently rediscovered for OPCode_schedAdmin after
already being documented once for OPCode_upgradeConfig.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
persistence.md's APPLY_CLEAN verdict only checks entity count and apply-
cursor position, not per-field completeness -- bug-1678 (job allocation
silently dropped on restart replay despite APPLY_CLEAN reporting healthy)
fell through exactly this gap. Documents the sharper invariant online-
apply and replay-apply must produce field-complete equivalent state, not
just matching counts/cursor, and adds it as a standing audit dimension
(#7) to xce-audit-invariants' controller dispatch list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Elevates the normal-mode/simulation-mode driver equivalence guarantee
from a one-line architectural description in CLAUDE.md to an enforced
steering constraint (scheduler-driver-equivalence.md), backed by
bug-1611/bug-1613 as concrete evidence of violation. Scopes the fix as
its own spec (scheduler-sim-decision-parity, requirements + design)
rather than reopening the closed scheduler-core spec or folding it
into scheduler-sim-repl-verbosity's explicitly narrower boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Full spec scaffold (requirements/design/research/tasks all approved,
ready_for_implementation=true) plus a roadmap backlog item recorded from
the dirty-content-fingerprint spec's task 4.3 findings (Makefile/meson
fingerprint-computation duplication risk).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Surfaced while investigating a user request to show total/current values
for GLOBAL/SHARED resources in the scheduler REPL's "info resource"
command -- researched and confirmed that concept doesn't exist anywhere
in this codebase (resourcemap config field dropped at build time, capnp
fields commented out, worker.c actively rejects XCE_RESF_SHARED resources
today). This spec already owns closing that gap; it just never made it
into the roadmap's backlog list.
- xce_test shell command support: was already implemented (xce-test-e2e-
primitives task 2.1, closed 2026-06-19) -- stale "not started" entry
corrected.
- scheduler-core tasks 2.1-2.6 _Verified: backfill: was already done
(commit d7b49690, 2026-08-08) -- corrects a status line I myself wrote
incorrectly earlier today by trusting the original 2026-08-07 finding's
wording without re-checking the file.
- job_ctrl direct-mutate-path investigation: completed. Real risk
confirmed (pre-commit state pushed live to scheduler + apiserver/clients
with no rollback on commit failure), scope wider than originally framed
(on_job_stop/on_job_resume/on_job_ckpt share the same pattern, fed by
ordinary traffic not just rare admin commands), plus a separate
read-lock-during-mutation bug found along the way. Not promoted to a
spec yet -- scope/framing corrected first per this session's decision.
- OP_upgradeConfig standby redirect verification: investigated. The
item's own premise doesn't match production (a standby never actually
returns a redirect for this opcode -- a documented, deliberate feature
gap, not a missing test); coverage confirmed already complete on both
the client mock side and a live 3-node regression test.
design.md's own "Open Questions/Risks" list (items 5 and 6) still described
the OPCode_schedAdmin admission gap and the tlog activation gap as open/
deferred, even though both were fixed earlier in this session -- marked
FIXED with pointers to the actual change. research.md's schedAdmin
decision record and its Risks & Mitigations bullet got the same treatment
(SUPERSEDES note, historical record preserved rather than rewritten).
tasks.md's 2026-08-07 GO/NO-GO follow-up list gained a status-update line:
follow-up 1 (tlog) is now fixed, follow-up 2 (fcfs.c/builtin_fcfs.c
traceability) is only partially closed (design.md text was already
correct; tasks 2.1-2.6 still lack _Verified: lines -- flagged, not
fixed here since I have no fresh verification evidence for those tasks).
roadmap.md's scheduler-core summary line corrected to attribute each gap
to the right source (2026-08-07 validate-impl vs today's separate
design-review pass) instead of conflating them.
Every early-exit branch in the worker_list scan either re-tested the same
node forever (bare `continue`) or returned out of the whole function on the
first DONE/EXITED worker, so job array elements past the head of the list
never got their on_job_start report. Capture `next` before touching each
node and always advance to it. Affects xcecd_virtual and real xcecd alike
(same DBUS_MANAGER-gated function).
Also records a roadmap follow-up (xcecd_virtual concurrent-job support,
MXJ > 1) discovered while investigating this bug.
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
These edits (three-thread picker correction in brief.md, record_dir
path-validation and kvmgr-audit requirements in requirements.md,
OPCode_schedAdmin folded into the peer-authentication-hardening
backlog item in roadmap.md) were made during earlier requirements
review and design work but left uncommitted. Reviewed for accuracy
against the completed implementation — all still correct, persisting
as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bug-1463: commit 94f50ce6 (bug-1453 CA-cert-plumbing removal) missed
GetJobOutput's dead tls.Config/loadRootCAs block in request.go, plus
peek.go's SetRootCAs call and serverInfo.CACert field access (the
field no longer exists on APIServerInfo) -- broke the xcecli build.
Removed all three dangling references. Also enabled NewAPIClient's
previously-commented-out InsecureSkipVerify TLS config, since no
xcecli method set any TLS config at all -- any https:// apiserver URL
would have failed cert verification, not just GetJobOutput. Matches
bug-1453's documented design (server-side TLS only, no peer
verification, mTLS deferred as P3 hardening).
Documented both the api-server (http+https) and controller
(server-side-cert-only, no client certs) TLS postures in
.kiro/steering/security.md, replacing a stale line claiming TLS is
handled at the network layer, plus a follow-up note for the planned
HAProxy TLS-termination reverse proxy once its cert type and
apiserver's own listener fate are decided.
