Persistent log of the resource-design discussion: the user's proposals
(generic supply map incl. provisioned supply; a resource provider proxy
between the controller and upstream providers; datasets in the map with
staging as data provisioning) with analysis, the user's 2026-10-11 input
(duration/decay must-have but future; RBAC in the apiserver, discuss later;
OCI containers via application profiles; CSI compatibility preferred;
GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items
and new questions N1-N4, C3. resource-model.md, the provisioner research,
roadmap and CLAUDE.md point at it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
resource-model.md: one resource type descriptor and provider contract over
three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler
only selects from its cached snapshot, providers run after dispatch; batch
stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims
as WAL entities with the resource-provisioner discipline; LSF/K8s mapping;
LLM training/serving needs; phases; nine pending user decisions.
New briefs: custom-resource-reporting, resource-instances,
resource-claims, gang-groups. Roadmap: long-term LLM training + serving
platform direction (features from LSF and Kubernetes), the four new specs,
updates to ten existing specs, discovery note. resource-distribution-mapping
becomes phase 1 (shared pools + descriptor fields).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Design constraint (tech.md "Process Creation"): the controller, scheduler,
xcecd and anything that starts libxce threads never call fork, vfork,
forkpty, daemon, system, popen or uv_spawn; children start with posix_spawn()
or a systemd unit. A process proven single-threaded may fork, marked
/* xce-fork-ok: <why> */. Tests, harness, autoscaler (not finalized; bug-1969)
and the Go frontend are out of scope.
- scripts/check-no-fork.py scans src/{controller,scheduler,lib,worker}
(comment/string/#if 0 aware); runs as meson test "no-fork rule" in the
lib/controller/scheduler/worker suites and in the pre-commit hook.
- worker: drop xcecd's unused fork-based worker start paths
(process_manager_start_process, process_manager_start_worker; xcecd
starts xcecd_worker over D-Bus) and a dead #if 0 daemonize fork.
- xcecd_worker's fork/uv_spawn/system calls are marked: its call graph
from main(), callbacks included, reaches no pthread_create or libuv
thread-pool call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New steering job-config-layering.md: queue = scheduling policy, application
profile = what the application needs and how it runs (resreq, task range,
execution settings, pre/post-exec, limits), project = resource ownership.
Only two interactions: the job overrides a profile attribute marked
default (fixed: not at all; select: only ANDed terms), and queue ceilings
or project quotas check effective values (reject if never satisfiable,
pend if full now). Replaces the simplified job > app > queue rules.
Queue pre_exec/post_exec are deprecated with queue res_req: the profile's
pre/post-exec fills the worker's existing outer layer, the job's own stays
the inner layer, allowed or forbidden by the profile.
job-config-merge and application-profile briefs and the roadmap follow;
CLAUDE.md lists the new steering file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gitea-sync.py fixes found on the first live run (bug-1855 follow-up):
- 204/empty-body replies (label DELETE) no longer crash in json.loads
- every request has a 60s timeout, so a stalled call fails instead of hanging
- spec.json sync state is inserted as a trailing "gitea" block instead of
re-serializing the file (15 hand-formatted spec.json files would reflow)
- implementation issue body labels generated vs approved correctly; the
validation body uses the same done-phase set as the close rules
Project memory now routes all buglog writes through the buglog skill:
CLAUDE.md key rule, a cerebrum Do-Not-Repeat entry (with the older
duplicate-id / json.dump / tags-vs-fix lessons tagged as superseded), and
xce-audit-invariants step 5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add harness to root CLAUDE.md's /xce-build component list and
Subdirectory Guidelines (pointing at src/harness/CLAUDE.md). Remove
build-deployment.md's false "controller always builds xce_test" claim
and add harness to its /xce-build command table, Meson Structure tree,
dependency-flow line, and worked compile-target example.
Review round 1 caught a third surviving instance of the same stale
claim in the bug-1565 illustrative paragraph, contradicting the
corrected text two paragraphs above it -- fixed to reference the
binaries controller actually produces (xce_controller,
controller_unit_test) instead of xce_test.
Task 3.6 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.
Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.
Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.
The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.
New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).
Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.
Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.
Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).
Full scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Prompted by rediscovering the same bug class in two places this
session: the controller's WAL already has this documented for one
mechanism (persistence.md's APPLY_CLEAN/bug-1678 section, xce-audit-
invariants controller dimension 7) but the scheduler's separate
snapshot save/load mechanism had the identical gap (job->allocs, this
session's scheduler-snapshot-job-allocation brief) with no equivalent
tracking. Closes that gap and makes the invariant self-reinforcing
going forward rather than something that has to be independently
rediscovered per persistence mechanism:
- xce-audit-invariants: new scheduler dimension 4 ("Field-complete
snapshot save/load parity"), sibling of controller dimension 7,
scoped to snapshot.c/snapshot_wire.c/snapshot_load.c.
- persistence.md: new "Snapshot Save/Load Field-Completeness
(Scheduler)" section cross-referencing the WAL-side bug-1678 writeup
and the new scheduler dimension.
- CLAUDE.md: new top-level Key Rules bullet stating the invariant
plainly, plus adding persistence.md to the steering-file pointer
table (it was missing).
- kiro-spec-design's design-review-gate.md (the mandatory pre-write
gate) and kiro-validate-design's design-review.md (the optional deep
review): new conditional review sections -- applies only when a
design touches a save/restore mechanism, requires enumerating every
persisted field's capture status rather than trusting an aggregate
"state looks right" check. Catches this class of gap at design time,
before implementation, rather than only via a later audit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
CLAUDE.md: replace the stale @.wolf/OPENWOLF.md import with a pointer to
the openwolf skill / rules file, matching how the project's OpenWolf
setup actually resolves the protocol now.
api-standards.md: add checklist item 6 for new primary-only opcodes --
confirm ha_dispatch_gate's -XCE_ERR_STANDBY rejection actually populates
a redirect via fill_redirect_reply(), not just that the client has a
generic redirect-follower. Directly informed by bug-1700, where the
identical gap was independently rediscovered for OPCode_schedAdmin after
already being documented once for OPCode_upgradeConfig.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
- create_lost_and_found(): provide hostattrs_t so makeWorker succeeds
- collect_queues/collect_workers (dataex.c): skip lost_and_found sentinel
so it is never pushed to apiserver or scheduler
- mark_worker_unavail(): reset hosttype/hostmodel to "unknown" and zero
all numeric attrs, then update resvals and slots — prevents stale
capacity from being visible while the host is down
- restore_worker(): apply the same attr reset on WAL recovery so
restored-but-unavailable workers advertise no capacity until reconnect;
also remove dead calloc() that was immediately overwritten by makeWorker
- Extract reset_hostattrs_unknown() helper to eliminate copy-paste
- Define XCE_LOST_AND_FOUND_NAME constant in config_repl.h; replace all
six raw "lost_and_found" string literals across config_repl.c and dataex.c
