Adds ug1_global_mem to limits.yaml: a permanent, real GLOBAL limit on
"mem" (a per-host resource, not "slots"), capping ug1's total
cluster-wide mem usage at 5GB. This is exactly the scenario bug-1790
fixed but only unit-tested until now -- no limit in the shipped example
config exercised a global cap on a non-slots per-host resource.
Live-verified via a fresh single-node deploy: 3 jobs submitted as user1
(ug1 member) each requesting rusage[mem=2000000000] (2GB) on queue
normal. Scheduler trace confirms ug1_global_mem correctly matches and,
for the first time, computes a real nonzero job-wide-total wanted value:
unsatisfy_ resource 36 total 5000000000.000000 inuse 4000000000.000000
uncomit 0.000000 wanted 2000000000.000000
unsatisfy_ unmet resource 36 (limit=ug1_global_mem) shortfall
1000000000.000000
The 3rd job (4GB inuse + 2GB wanted = 6GB > 5GB cap) correctly PENDed
until an earlier job's mem freed up. Deploy config validated with zero
new warnings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Add workerResyncWindowMs/workerResyncTimeoutMs to xce.schema.json
(minimum:0, matching the optional-scalar-field precedent -- these
aren't `required` with a floor like heartbeatInterval, so minimum:1
would be wrong) and to mapping.py's YAML->capnp field table. Regenerate
both golden config-build fixtures (task 1.1 grew ClusterConfig's wire
layout by 2 fields; full/expected.bin and minimal/expected.bin each
grew by exactly 8 bytes). full's fixture YAML now sets real values
(900000ms/15000ms); minimal deliberately left untouched (no xce.yaml)
per its own locked-down "hosts+queues only" boundary test -- its
expected.bin still regenerates to reflect the new default-0 fields.
Two new tests assert specific decoded values, not just "build succeeds".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds `xceadmin config dump CLUSTER_BIN_PATH`, the reverse of `config
build` -- decodes a serialized ClusterConfig back to human-readable text
so a built config can be inspected without a throwaway pycapnp script.
Motivated directly by how bug-1556 (hosts.yaml's max_slot: -1 auto-detect
sentinel silently becoming mxj=0) was actually found: by hand-decoding a
live cluster.bin. The dump renders the same sentinel as
"max_slot=-1 (auto-detect from CPU count)", not the raw uint32.
Also adds parse_size_or_number() to xcecommon.mapping so a queue/host's
sched_stop threshold on a byte-valued resource (maxmem/mem/tmp/maxswap/
maxtmp) can be written as a human-friendly size string ("500MB", "2GB")
instead of a raw byte count; a plain number still passes through
unchanged for non-byte resources like load averages. hosts.schema.json
and queues.schema.json's sched_stop schema gains a matching SizeOrNumber
type. dump_cluster_config() humanizes the same byte-valued thresholds
back on the way out (_format_bytes_human).
New ClusterBinFileError covers every local cluster.bin read/decode
failure mode without letting a raw OSError/capnp.KjException escape the
CLI. RESOURCE_VALTYPE/RESOURCE_ORDTYPE/RESOURCE_FLAG_* made public in
mapping.py so dump_cluster_config can reuse the same int<->name tables
build_cluster_config already uses, keeping both directions in sync by
construction.
256 xceadmin tests pass (incl. new test_config_dump.py and the
parse_size_or_number/SizeOrNumber cases in test_mapping.py/test_cli.py),
verified via `xce-build.sh xceadmin`'s full pinned out-of-tree install
+ pytest run before this commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Simplification per review feedback on e3b3ffa: a generic schema-driven
unit field touched the capnp wire schema (2 structs), C structs
(xce_resource_t/xce_resval_t), resobj.c's deep-copy/destroy, opget.c's
reply-build lookup, resources.yaml/resources.schema.json, and mapping.py
-- real cross-cutting cost for a mechanism only one resource (maxmem)
actually needs today.
Reverted all of that. The controller CLI (test.c) now just knows
"maxmem" is bytes-valued via a small explicit is_bytes_valued_resource()
name list, and adaptively picks the best-fit human unit at display time
(format_bytes_human, unchanged) -- same observable behavior (resval/
workerlist render maxmem as e.g. "30.90 GB"), zero wire schema surface.
Regenerated the xceadmin golden fixture back to its pre-unit-field size
(5304 bytes, exactly matching the size right after the mxj-only fix,
confirming a clean, complete removal).
Verified: 1095/1095 controller_unit_test (3 runs; one hit the
pre-existing flaky segfault already logged as bug-1553, unrelated),
228/228 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three bugs found while diagnosing a live controller log, all in the
worker resource-value path:
1. Every host relying on the documented hosts.yaml auto-detect default
(no max_slot, meaning "use CPU count") got 0 scheduling slots,
permanently blocking the scheduler from placing jobs on it. mxj is
UInt32 on the wire and can't carry the schema's -1 sentinel; mapping.py
worked around this by leaving it unset, which defaults to 0 -- and 0 is
never < 0, so apply_join_state()'s auto-detect fallback never fired.
Fixed by encoding the sentinel as 0xFFFFFFFF, which the controller's
`int mxj` field reinterprets back to -1 via its existing implicit
uint32_t->int conversion. No C-side or wire schema change needed for
this part; the fallback itself was already correct but had zero test
coverage for a negative mxj, now added.
2. update_resvals()/update_resval_list() logged the failure reason via
PERROR, which reads process-global errno -- but update_resval()'s
failure paths return raw negative codes directly and never touch
errno, so the controller log filled with misleading "update_resval:
Success" / "update_resval: Resource temporarily unavailable" lines
instead of the real reason. Fixed to decode the actual rc via
xce_strerror_r and name the failing resource.
3. maxmem is canonically stored/wired as raw bytes but had no unit
metadata, so resval/workerlist printed it as a bare number that read
as garbage (e.g. workerlist's fixed-width columns visually smeared a
multi-GB byte count across the MaxCPU/MaxMem headers). Added an
optional `unit` field to the Resource/ResourceVal capnp structs (and
xce_resource_t/xce_resval_t/resources.yaml) so resval and workerlist
can render a human-readable value (e.g. "30.90 GB") while the stored
value stays raw bytes -- display-only, every other resource unaffected.
Regenerated the xceadmin golden fixture (fixtures/full/expected.bin)
twice: once for the mxj sentinel encoding change, once for the new
Resource.unit field growing every resource's wire size by one pointer
slot. Both diffs verified byte-for-byte isolated to exactly the expected
change before regenerating.
Verified: 1095/1095 controller_unit_test (stable across repeated runs),
229/229 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Two related findings logged to .wolf/buglog.json but deliberately not
fixed here (out of scope per discussion): a pre-existing flaky segfault
in the combined controller_unit_test binary, and maxswap/maxtmp being
hardcoded stub values on the worker side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cross-reference warnings ("references undefined entry") were genuine
gaps in the example configs, not validator bugs: src/configs/hosts.yaml
had a typo'd resource ref (mg -> mem) and referenced a nonexistent
host3/hostgroup pair; queues.yaml referenced hgroup1/hgroup2/ugroup1
instead of the real hg1/ug1 names; resources.yaml was missing the
r1m/r15s/tmp built-in worker load-index resources
(worker/src/resource_sync.c's init_builtin_load_indices()). Added the
missing host3 host, hg2 hostgroup, ug2 usergroup, and three resource
entries; fixed the typo'd refs.
"schema-valid but has no ClusterConfig destination" warnings for
resources.yaml's type/order/consumable/release fields were a real gap
in xcecommon/mapping.py's _build_resource, not an inherent limitation
-- the encoding (XCE_VAL_*/xce_order_t/XCE_RESF_CONSUMABLE/
XCE_RESF_RELEASE) is documented in include/xce/types.h and actively
read by the scheduler; the old docstring's claim that it was
undocumented was wrong. Only resourcemap remains genuinely unmapped
(its capnp destination fields are commented out).
Updated test_mapping.py/test_config_build.py for the new mapping
behavior and regenerated the full golden fixture's expected.bin
(legitimate representation-drift from the mapping fix, not a bug).
xcesched_push_configs() sent two hardcoded limit1/limits2 test
literals to the scheduler (behind a #if 0/FIXME "we use builtin limit
currently"), ignoring whatever was actually loaded from cluster_config.
Migrate those literals into limits.yaml as real entries and wire
conf4sched->nlimits/limits from c->myconf->cluster_config directly.
cluster_init_phase2 also downgrades its "cluster_config not loaded"
ERROR to INFO — a standby legitimately reaching phase2 before the
primary's config-push has delivered bytes yet is normal cold-boot
behavior, not a failure; it already matches load_primary_config's
INFO-level framing for the same condition.
design.md, research.md (new), brief.md and requirements.md (updated
during spec authoring: Req 1.6 resolved/superseded by bundling, Req
5.5 added, Req 4.3/4.4 extended for limits.yaml) had been sitting
uncommitted since before implementation began -- the entire 18-task
implementation was built against these documents without them ever
landing in git.
src/configs/*.yaml + schema/*.schema.json: the example config
directory and bundled JSON schemas most of the test suite reads as
"real repo fixtures" (test_config_build.py, test_mapping.py,
test_validation.py, etc.) were likewise never committed -- those
tests would silently skip (skipif guards) rather than run for anyone
else checking out this branch.
