feat(scheduler): reconcile mem reservation against real usage

Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.

Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.

The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.

New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).

Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.

Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.

Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).

Full scheduler suite: 55/55 passing, zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA

2026-08-31 07:02:32 +00:00

7.1 KiB

Raw Blame History

OpenWolf

This project uses OpenWolf for context management. The always-on rules live in .claude/rules/openwolf.md; the hooks handle bookkeeping (anatomy index, memory log, read tracking) automatically.

For the full operating protocol (session handoff, memory discipline, bug logging), load the openwolf skill, or read .wolf/OPENWOLF.md. Regenerate the session handoff with /handoff.

XCE Cluster Management System

Project Overview

XCE is an HPC cluster management system — Go API/CLI tier fronting C daemons (controller, worker, scheduler). Full context in .kiro/steering/product.md.

Engineering Guide

For details on any topic below, read the corresponding steering file in .kiro/steering/. CLAUDE.md provides pointers, not duplicates.

Topic	Steering File
Project purpose, HA phases, design decisions	`product.md`
Technology stack, build systems, key decisions	`tech.md`
Directory layout, naming conventions, test org	`structure.md`
Build commands, Nix flake, deploy, AppImage	`build-deployment.md`
REST handler pattern, daemon opcodes, Cap'n Proto	`api-standards.md`
HA protocol, async I/O, Raft election, heartbeat	`ha-async-io.md`
TDD rules, C + Go test patterns, coverage targets	`testing.md`
Error handling: Go ReturnError, C errnox, zap logging	`error-handling.md`
RBAC, JWT, auth backends, input validation, secrets	`security.md`
WAL durability, recovery, save/restore field-completeness	`persistence.md`
Real vs. virtual resources, reservation-vs-usage accounting (mem)	`resource-accounting.md`

Key Rules (from steering — summarized for quick reference)

TDD mandatory: red → green → refactor; test in testsuite/ before C changes, _test.go before Go changes
Async I/O: delaymgr is trigger-only and must never block (no connect(), no file I/O) — it drives heartbeats/election timeouts, so blocking there causes spurious failover. Long/unbounded blocking goes on the slow lane; bounded blocking in vtask handlers is permitted only under an admission reservation (COMMIT_WAIT_RESERVED). xce_request/xce_reply are always non-blocking. See ha-async-io.md
Connections are long-lived: closing signals RDHUP failover. Inbound connections are vote-only; outbound carry all state push
Middleware chain (non-negotiable): WhitelistURL → WhitelistIP → RateLimit → JWT → CORS → RBAC
Error pattern: ReturnError(code, msg, nil); return — early return on every error
Build: /xce-build [component] — debug + ASAN + tests. See build-deployment.md for full commands
Save/restore field-completeness: any persistence mechanism (WAL fact, scheduler snapshot, checkpoint) must capture every field of the entity it claims to persist — a state that "recovers correctly" while silently missing a field it depends on (e.g. job.state correct but job.allocs lost) is a bug, not an acceptable simplification, unless the omission is explicitly documented as a scope decision (see persistence.md's "Snapshot Save/Load Field-Completeness" + APPLY_CLEAN/bug-1678 sections). When adding or extending save/restore logic, verify the field round-trips through BOTH the write side and every read/apply seam, and run /xce-audit-invariants (controller dimension 7 / scheduler dimension 4) rather than assuming a passing test suite already proves it — APPLY_CLEAN-style verdicts check row counts and cursor position, not per-field completeness

Key Files

src/frontend/Makefile — primary build/test orchestration
src/frontend/apiserver/cmd/main.go — server startup, init sequence
src/frontend/apiserver/router/routers.go — all routes and middleware chain
src/frontend/apiserver/controllers/rbac_check.go — RBAC loading and enforcement
src/frontend/apiserver/config/rbac.yaml — role/permission definitions
src/frontend/apiserver/config/auth.yaml — auth provider selection

Subdirectory Guidelines

src/frontend/ — See src/frontend/CLAUDE.md
src/controller/ — See src/controller/CLAUDE.md
src/worker/ — See src/worker/CLAUDE.md
src/scheduler/ — See src/scheduler/CLAUDE.md
src/lib/ — See src/lib/CLAUDE.md

Agentic SDLC and Spec-Driven Development

Kiro-style Spec-Driven Development on an agentic SDLC

Project Context

Paths

Steering: .kiro/steering/
Specs: .kiro/specs/

Steering vs Specification

Steering (.kiro/steering/) - Guide AI with project-wide rules and context Specs (.kiro/specs/) - Formalize development process for individual features

Active Specifications

Check .kiro/specs/ for active specifications
Use /kiro-spec-status [feature-name] to check progress

Development Guidelines

Think in English, generate responses in English. All Markdown content written to project files (e.g., requirements.md, design.md, tasks.md, research.md, validation reports) MUST be written in the target language configured for this specification (see spec.json.language).

Minimal Workflow

Phase 0 (optional): /kiro-steering, /kiro-steering-custom
Phase 1 (Specification):
- /kiro-spec-init "description" — initialize spec scaffold
- /kiro-spec-quick "description" — fast single-pass spec (requirements + design + tasks, no approval gates)
- /kiro-spec-batch — generate specs for all features in a roadmap in parallel
- /kiro-spec-requirements {feature}
- /kiro-validate-gap {feature} (optional: for existing codebase)
- /kiro-spec-design {feature} [-y]
- /kiro-validate-design {feature} (optional: design review)
- /kiro-spec-tasks {feature} [-y]
Phase 2 (Implementation): /kiro-impl {feature} [tasks]
- /kiro-validate-impl {feature} (optional: after implementation)
- /kiro-verify-completion {feature} — validate all acceptance criteria are met
Debugging: /kiro-debug {feature} — systematic debugging with spec context
Code review: /kiro-review — review implementation against spec requirements
Progress check: /kiro-spec-status {feature} (use anytime)

Development Rules

3-phase approval workflow: Requirements → Design → Tasks → Implementation
Human review required each phase; use -y only for intentional fast-track
Use /kiro-spec-quick for small, well-understood features; full workflow for anything touching HA/persistence/RBAC
Keep steering current and verify alignment with /kiro-spec-status
Follow the user's instructions precisely, and within that scope act autonomously: gather the necessary context and complete the requested work end-to-end in this run, asking questions only when essential information is missing or the instructions are critically ambiguous.

Steering Configuration

Load entire .kiro/steering/ as project memory — all .md files are authoritative
Core: product.md, tech.md, structure.md
Custom: testing.md, error-handling.md, security.md, ha-async-io.md, api-standards.md, build-deployment.md, config-design.md, persistence.md, resource-accounting.md, roadmap.md
Add new custom files via /kiro-steering-custom; sync existing via /kiro-steering

7.1 KiB Raw Blame History