The Aug 30 scheduler-framework merge (3b6f53bb) dropped .wolf/OPENWOLF.md,
anatomy.md and memory.md (Rongsong's tree carries no .wolf state); CLAUDE.md
@-includes OPENWOLF.md, so the session protocol has been silently missing
since. Restore the protocol verbatim (ba4e5fd2) and commit the current
hardening-plan/Phase 0/1/HA-analysis bookkeeping.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Phase 1 of scheduler-hardening-plan.md: mechanically locks the bug-1521
contract that 91c3fb7f established but which lived only in code comments:
the allocator thread is the ONLY writer of a schedobj's reason fields;
the picker thread never touches reasons.
test_reason_ownership_audit.c (in the sched_unit_test aggregate) enforces
this in three layers:
- a whole-tree scan of every src/scheduler source (fixed 51-file list) for
schedobj-shaped reason ownership lines (obj->reason / schedobj->reason /
primary->reason / cobj->reason / compound->reason combined with
MOVE_PTR( / free_PendReason_ptr( / calloc( / a bare = store), checked
against an OWNERSHIP_SITES allowlist whose fragments must still exist in
their files (so the list cannot rot silently);
- a dedicated scan of sched_job_picker()'s brace-matched function body
rejecting ANY reason ownership inside the picker;
- detector self-tests proving it fires on the historical violation shapes
and stays quiet on sanctioned reads, ctx->reason (the sched_ctx_t local
shuttle), construction =NULL, and comment lines (skipped by the scans --
the contract's own documentation names the write shape in prose at
scheduler.c:496 / fcfs.c:243).
Red proof: a temporary MOVE_PTR(job->schedobj->reason, ...) injected into
the picker loop trips BOTH the file scan and the picker-body scan; the
injection is reverted and the suite is green.
.kiro/steering/scheduler-reason-ownership.md formalizes the discipline
(who allocates, who frees, which thread, MOVE_PTR semantics, the
ctx->reason exemption) for review use.
Full scheduler suite: 102/102 OK.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Phase 0 of the scheduler hardening plan is done:
- 0.1 (three-path skip symmetry): landed as test_res_account_symmetry.c
(b20caf8d), mutation-injection red-proven.
- 0.2 (rejoin trigger contract): audit found the contract fully covered
by test_scheduler_decision_publish_retry.c's five rejoin tests plus
test_scheduler_connect.c's no-io callback scan -- documented in the
plan instead of adding duplicate tests.
- 0.3 (two-round zero drift): folded into 0.1's
test_two_rounds_zero_drift_with_reconciling_worker, same shape.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Phase 0 of scheduler-hardening-plan.md: a cross-file structural regression
test locking the invariant whose three observed violations were bug-1799
(charge without reset-skip), bug-1801 (the complement) and bug-1814
(rusage delta without the skip): every w->resInUse/core->sharedInUse-
mutating path shares ONE identical skip condition for UNAVAIL/RECONCILING
workers.
test_res_account_symmetry.c drives all three paths in one core per round
-- core_reset_workers() (core_session.c, bug-1751 skip),
sched_charge_account()/sched_update_account() (resource.c, bug-1801 skip)
and sched_reconcile_host_resource_usage() (resource.c, bug-1814 skip) --
using the real sched_account_jobs() from scheduler.c, and asserts the
accounts do not drift across consecutive rounds. A per-path test cannot
see asymmetry by construction; bug-1799's live shape was asymmetry
compounding across rounds (resInUse grew one full charge per round for
two weeks).
Three cases: two-round zero drift with a RECONCILING worker (B frozen at
its seeded value, both host entries named by the job's account so the
skips are active decisions, not vacuous); UNAVAIL worker plus a real
usage climb 2.0->5.0->0.0 proving the reconcile delta still applies on
the healthy host while the excluded host's account-entry used baseline
never advances (bug-1814's half-damage); and a status-clear round
proving the freeze is per-round status-gated, never permanent.
Link technique: the separate-compiled-sources shape of
sched_trace_error_paths_test -- the test white-box #includes
core.c/core_session.c/resource.c/schedobj.c + the four hooked trace
sources, and meson compiles every other scheduler_srcs entry (minus
main.c) as its own TU, which resolves sched_account_jobs() from
scheduler.c's own object without white-box including scheduler.c (and
without the plugin_* stubs test_resource_accounting.c needs: real
plugin.c is one of those TUs).
Red proof: temporarily removing resource.c's bug-1801 skip trips the
zero-drift assertion immediately (2.5 -> 4.5, exactly one extra charge
per round -- bug-1799's live shape); mutation reverted, suite green.
Full scheduler suite: 102/102 OK.
Co-Authored-By: Claude Code <noreply@anthropic.com>
bug-1980: ParseCmdArgs treated any non-terminal stdin as a potential job
script and io.ReadAll'ed it before considering the command args, so a
pipe that stayed open without writing data or EOF blocked submit
forever — no error, no RPC, no job. Found as a day-old hung process in
the user's process list (an esd-recover session's own submit probe).
bsub semantics the CLI claims to follow are the reverse: stdin is the
job script only when NO command is given on the command line. Gate the
stdin read on len(args) == 0; an explicit command wins and stdin is
never touched (also removes the ambiguity of stdin bytes being silently
reinterpreted as a config script next to an explicit command).
- TestParseCmdArgs_ExplicitCommandIgnoresHeldOpenStdin: held-open pipe
+ explicit args; RED pre-fix (blocked in io.ReadAll), GREEN post-fix.
- Stdin-script path (no args) verified live: script → submitted → DONE.
- Live: submit with held-open stdin went rc=124/no-job to instant
submit → RUN → DONE on t1.
- xcecli go test ./... 11/11 packages ok.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Live-verified on t1 (2026-10-29): kill -9 xceesd with 3 RUN license jobs
holding markers, restart, and confirm recovery end to end.
- §10.1: recover semantics — xceesd is a stateless collector; fresh
restart joins, skips full re-push (0/1 sources, no cached value), and
the first poll round pushes the collector script's live value, which
re-aligns the pool in one shot. Correctness rests on the script being
liveness-based (§5.3).
- §1.2: esd outage is the unbounded form of the exposure window — the
controller freezes on the last pushed value and STALE is observation
only, never an admission input.
- §11.5 (new, scenario 4): full E2E recipe with pass criteria, incl.
the ops pitfall found while testing — a PEND during the outage may be
a slots limit (limit1 per-user cap) rather than the license pool;
check pending_reason via the apiserver JSON (CLI shows "-").
Co-Authored-By: Claude Code <noreply@anthropic.com>
bug-1979: test.c never included resobj.h, so makeResource() was an
implicitly declared function in this TU -- its 64-bit pointer return was
truncated to int (0x50b000000118 -> 0x118) and sign-extended back.
Dormant since E2 Phase 1 (the test only TEST_ASSERT_NOT_NULL'd it and
re-derived the real pointer via key_find); the leak-fix teardown's
dereference(made) was its first real use and SEGV'd under
ASAN_OPTIONS=detect_leaks=1 (the xce-build.sh all direct-run step).
- include resobj.h: makeResource now declared with its true prototype
- unmeet: memset-only -- sched_check_shared_satisfy_() xvec_idval_init()s
it itself; the pre-init leaked the vector it replaced (2x4B per call)
- uncomitted: single xvec_init/xvec_fini pair (was init'd twice, fini'd once)
- teardown: collect-then-dereference the refcount-1 resource object, then
destroy(resmgr), then xstring_table_fini() -- mirrors xce_sched_core_fini()
(core.c:405)
Verified: ASAN_OPTIONS=detect_leaks=1 direct run rc=0, 17/17 PASS, zero
leak reports; xce-build.sh all green (137/137 suites + manual sched_test
+ Go tests).
Co-Authored-By: Claude Code <noreply@anthropic.com>
A controller restart dropped xceesd's resval pushes forever: the fresh
endpoint is created by handle_connection() with priv->role =
HostRole_client, and only OPCode_join sets HostRole_extres -- but the
old esd_on_reconnect cleared only g_needs_full_push, so the daemon
re-PUSHED onto an un-joined endpoint and every push landed in
do_update_resvals's neither-branch, refused -EPERM forever (live
sighting: pushes_ok frozen, rc=-1 every 10s).
Mirror the worker's needs_rejoin contract: the reconnect handler now
sets BOTH g_needs_rejoin and g_needs_full_push; the main loop consumes
the re-join first (xcecli_join(HostRole_extres) on the main thread,
outside the failover lock -- the xcecd_on_reconnect no-I/O constraint),
retrying next tick on failure, then does the full re-push.
Live t1/t2 proof: controller restart -> "esd: re-joined after
reconnect (rc=0)" + "full re-push: 1/1 source(s)" + extlic back in
`xcecli resources distribution list`. Two new white-box unit tests pin
the both-flags contract (13/13 in the esd suite).
Co-Authored-By: Claude Code <noreply@anthropic.com>
The external-license-resources E2 fixture builder crr_build_config
registers five resources and four distributions, writing resources[4],
dists[3] and vals[4]. All four caller test functions declared only
resources[4]/dists[3]/vals[4], so every build_config call wrote past
the caller's stack arrays -- deterministic ASAN stack-buffer-overflow,
the whole controller suite (137 fixtures) red since 2fcacac2.
Resize the four caller arrays to match the builder's sizing
(resources[5]/dists[4]/vals[5]) with a bug-1977 comment on the sizing
contract. No production code touched.
Verified: src/tools/xce-build.sh controller (asan) 6/6 OK, controller
suite 91s green; buglog record .wolf/bugs/bug-1977.md.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Per-host user-script load indices (LSF elim model): a config-declared
script collector (external_resources: [{name, command, interval}]) in
xcecd registers getters into resource_sync's load-index registry, so
user-defined dynamic resources flow the existing resval chain
(collect -> xcecli_update_resvals -> controller -> scheduler) without
any new daemon.
- elim.c: libcyaml config parse (fail-loud at load), posix_spawn script
runner with kill-watchdog timeout (mirrors esd_script.c, no fork in
xcecd), first-line output parse (bare number or "<name> <value>"),
per-source interval cache (getter serves last value until next_run),
fail-open keep-last-value on script failure
- resource_sync.c: fix two init-order traps exposed by external
registration -- init_builtin_load_indices no longer re-callocs a
populated g_load_indices array (clobbered entries -> strcmp garbage
-> segfault), and register self-allocates when called before the
first collect (capacity-0 trap); new public
resource_sync_ensure_builtin_load_indices() so elim_register_sources
can register before lazy builtin init and builtin-name shadowing is
a clean -EEXIST at startup
- main.c: xcecd -c/--config option (elim config file); ownership
main.c -> xcecd_init -> resource_sync registration, with rollback on
init failure and cleanup in xcecd_cleanup
- tests: 18-case white-box elim_test (config/parse/getter-cache/
register/unregister/script-runner incl. timeout no-zombie), plus
stubs for the startup-order test
- t1/t2/t3 E2E: per-host dynres1 values visible in CLI (11/22/33 from
each node's own script); job with rusage[dynres1=15] RUNs on t2
(22 >= 15, t1=11 correctly skipped), RUN->DONE; over-ask
rusage[dynres1=50] stays PEND; second saturating job overflows to
t3; kill frees the reservation and a new job re-fills t2
Note: rusage clauses use square-bracket syntax rusage[...]; the
LSF-style paren form rusage(...) is parsed as a select expression and
rejected by the select grammar (observed live, pre-existing).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two gaps in the Phase 3 regression net:
1. test_core_events.c: test_job_update_run_adoption_carries_shared_allocs
-- scheduler-side counterpart of bug-1976. A JOB_UPDATE marking a job
RUN whose jalloc carries shared-pool records (the per-project limit
charge + the external resource itself) must rebuild job->accounts
with accounts_shared mirroring them, or the next sched_account_jobs
pass never charges the limit vres and every global limit admits
forever. Also pins the payload ownership move (updated.allocs == NULL).
2. test_limit_project_scope.c: test_queue_scope_per_queue_instance_names
-- the per_queue shape of limits.yaml's extlic_qdist: queues:-scoped
limit slices its usage instance per queue ("limit=<name>/q=<queue>")
exactly as per_project slices by project ("/p="); cap lands at the
instance's own vres idx; out-of-scope queues match nothing. Uses the
same heap group-config lifetime contract as the project shape, so
free_limit_config() gained queues handling (and NULL-guarded the
projects group -- per_queue configs carry projects == NULL, which the
unconditional dereference crashed on; deterministic under
MALLOC_PERTURB_, every run).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Strategy layer (spec .kiro/specs/external-license-resources design.md 3.5/3.6):
- limits.yaml samples: extlic_share (per-project, projA/projB, cap 3.0) and
extlic_qdist (per-queue, normal/night, cap 4.0) on the externally reported
extlic pool -- instances "limit=extlic_share/p=<proj>" and
"limit=extlic_qdist/q=<queue>" under the existing limit vres mechanism.
Multi-daemon redundancy is free (resval last-writer-wins); manual override
REST deliberately out of scope.
- White-box scheduler test (test_limit_project_scope.c): project-scope
match + vres instantiation, per-project instance fan-out with caps,
unknown underlying resource failing clean. 3/3 under ASAN.
Controller fix (the Phase 3 E2E blocker, TDD red->green):
- dispatch_a_job_() rebuilt j->job.allocs from the scheduler's dispatch
payload but migrated only the per-host records -- allocs_shared (the
limit vres charge among them) was dropped, and its payload pointer
leaked. That rebuilt record is what sched_job_update_fun() re-sends to
the scheduler as ground truth (the JOB_UPDATE adoption there rebuilds
job->accounts from it), so sharedInUse never accumulated the vres
charge: unsatisfy_ admitted past every global limit forever.
Fix: MOVE the shared records alongside the host records (test
test_dc_success_migrates_shared_allocs).
t1 E2E (4 licenses, extlic_share projA/projB cap 3.0): 3 projA RUN, 4th
projA PEND, projB unaffected (own per-project instance); killing one
RUN job re-admits the pending projA; external free pool and per-project
shares enforced independently; all daemons healthy throughout.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.
Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
update_resval_list(c, NULL, ...) — host=="" entries route through
update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
— EEXIST-tolerant so definitions added by an activation register on a
RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
fields (valtype/ordtype/flags/interval) in place so a corrected config
heals a stale registration (bug-1974); seed_sharedinst_from_config skips
EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
capacity while the daemon is down)
Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
snapshot_load's host=="" precedent — previously WARN-skipped as unknown
worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
(F >= uncomit + wanted; F is already net of our checkouts)
xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
nothing and reconnects only fire inside a push — after a controller
restart the daemon sat on a dead connection forever; the main loop now
forces a full re-push when xcecli_is_connection_broken() (bug-1760
predicate) latches
Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
xceadmin tests
Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.
Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.
Co-Authored-By: Claude Code <noreply@anthropic.com>
rusage collection (merge cb0c7fe9 regression, root cause of 0 counters):
- collect_job_rusages builds the systemd unit name with
make_job_sd_unit_name (four-segment fmt_jobid_ex form) instead of a bare
%lu of the packed uint64, so the D-Bus property lookups hit the real
per-job unit again. E2E on t1: DONE job reports max_mem/avg_mem/cpu_time
non-zero (previously all 0).
XCE_JOBID written identity:
- exdata sets XCE_JOBID via fmt_jobid (four-segment
cluster-apiserver-timestamp-seq) instead of a bare packed decimal;
XCE_JOBINDEX handling unchanged.
- xrun parses XCE_JOBID with parse_jobid (accepts the four-segment form);
a bare decimal no longer round-trips through the written identity.
- E2E on t1: job sees XCE_JOBID=<submit id>; xrun remote-start round-trip
to another host matches the job.
select WARN (silent degradation):
- parse_resreq_primary WARNs when a select references an unregistered
resource ("...references an unregistered resource -- dropping the select
and skipping any rusage[] clause of this primary") instead of failing
silently; scheduler log shows the WARN on t1 and the job schedules
without the constraint.
CLI env handling (unit-tested; subEnvVars is not flag-bound so this is
not reachable from the production CLI):
- getEnvironmentVariables keeps B=b=c (split at first '=') and warns +
drops a bare key or an '=value' fragment instead of silently mis-parsing.
Docs:
- src/docs/xce.md: document the dead-clause semantics (select/rusage
primary drop behavior) matching the WARN.
bug-1972 (test-only UAF): test_job_live_gc asserted job destruction by
reading refcount from the freed job; rewrite the assertion to observe
destruction without touching freed memory. controller_unit_test is green
again (1354 tests).
Regenerate 3 repl_trace_baseline fixtures whose baselines shifted with the
select-WARN logging; replay tests green.
Spec naming: external-license-resources design.md/brief.md fix the
self-referential daemon-name note (old drafts said xceextresd/extresd).
Verified on t1: rusage_burn job (counters non-zero in DONE), echo_env job
(four-segment XCE_JOBID), select[nosuchres] job (WARN + schedules),
xrun t2 remote-start round-trip, CLI env unit tests, xcelib 762 /
controller 1354 / worker exdata 8+11 / sched repl_trace 4+14+4+10+17
tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
LSF License Scheduler-like external resource management: xceextresd
daemon (flexlm/script collectors) joins the controller with a new
HostRole member and pushes server-free counts through the existing
resval pipeline (wire unchanged apart from HostRole); EXTERNAL-shared
resources skip the inuse term in the scheduler's satisfy check
(Slurm-deficit-equivalent). Covers the elim-compat survey (worker
resource_sync load-index registry activation, separate D-size item),
the license-merge attribution layering (dispatch-ledger quotas +
tiered evidence + residual bucket), and the mergeability review
against Rongsong's unified resource model specs (P9 resolution
proposal: per-resource accounting mode, one writer each).
Proposal stage only: no EARS requirements/tasks yet; the
cross-team items (P9 mode split, project-config cluster-level pool,
peer-auth allowlist) are recorded in design.md section 3.6 for
discussion. Not to be merged to Rongsong's repo.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The LLM server on this host is restarted by hand (llmserver relaunches),
so the observe loop must surface availability instead of discovering an
outage only when a trigger happens to fire — on a quiet cluster that gap
used to be hours. Each scan round now pings the primary (1-token chat
call, single attempt, 5s cap — a probe reports state, it is not a
business call): a flip logs "llm DOWN: <cause>" once and "llm UP again
after <d> down" on recovery, and every clean-scan line carries llm=up/
down/off. Echo-only mode and providers without a Ping method are never
probed and report off/assumed-up — no state is claimed that we do not
know.
Co-Authored-By: Claude Code <noreply@anthropic.com>
`xce id` printed empty name/status: two breaks in the chain.
1. CLI decode: the clusters API returns store-backed rows in the OrderMap
wire format ({"Fields":[...],"Data":{...}}), but ClusterResponse.Rows
([]ClusterInfo) had no custom UnmarshalJSON, so every row landed
all-empty. Same gap array_stats hit; fixed the same way
(UnmarshalOrderMapRow).
2. Server data: the cluster row only comes from OnClusterUpdated (config
push), which never writes Status or ControllerName -- both stayed at
the store zero value (status 0 = DISCONNECT) forever even on a
healthy cluster. The controller's real identity/health snapshot
already exists on the wire (CtrlStatus reply, xce_ctrlstatus_t); the
apiserver just never asked for it.
Fix:
- backend.FetchCtrlStatus(): CGO wrapper of xcecli_ctrlstatus through
callBackend (failover + circuit breaker like every other read),
decoded into ClusterCtrlStatus and freed via free_CtrlStatus_ptr
(bug-1401 leak contract); no-CGO stub added alongside the others.
- GetClustersInfo merges the snapshot into each row before rendering:
cluster_name must match the responder's own (a stale-named row never
claims another cluster's identity), controller_name <- primaryName,
status <- CONNECTED. A backend that doesn't answer keeps the store's
last-known state instead of failing the query.
- CLI ClusterInfo gains the OrderMap-aware UnmarshalJSON.
t1 live: `xce id` now shows name <eda1> / status <CONNECTED> /
controller t1; submit->run->DONE job verified with all daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Terminal jobs (DONE/EXIT/PERR) were only filtered from checkpoints and
WAL replay (job_should_retain, finished_retain_s=3600) — the live jobmgr
never dropped them, so joblist and the scheduler's core->jobs view grew
without bound across a controller's uptime (N4).
Three pieces:
1. kvmgr delete path (src/lib/kvmgr.c): xkvmgr_type had .del_key=NULL
and .vclear=NULL, so key_del()/vclear() on a kvmgr only returned
-EINVAL. New xkvmgr_del_key (key_find to locate the slot idx ->
key_del — kvtab's val_killer is NULL so the idx scalar is untouched ->
vclear(data, idx)) and xkvmgr_vclear forwarder, wired into the
typeinfo. A missing key propagates -ENOENT (same contract as
xkvmgr_find).
2. Controller periodic sweep (job.c): job_live_gc_sweep — primary-only
(standbys rebuild their live table from WAL replay; deleting there
diverges), collects !job_should_retain jobs under the jobmgr READ
lock with reference() (check_workers_timeout collect pattern), then
per-job WRITE-locked key_del. TWO dereferences per purge: the
collection reference and the table's own makeJob reference — the
jobmgr's datakiller (job_obj_killer) is a deliberate no-op, so
key_del transfers the table's ownership duty to the caller; one
dereference would leak at refcount 1 forever (the very accumulation
this sweep drains). Registered by cluster_start_job_live_gc with
ha_start_primary_heartbeat's idempotent static-arm pattern +
delaymgr.running guard (60s, runcount=-1).
3. Registration on both promotion tails (ha.c): raft road after
ha_start_primary_heartbeat and the non-raft ha_promote_to_primary.
NOT cluster_start — it is orphaned on the HA myapp_run path.
Safety: apply_committed_entry's jobstate group and WAL_JOB_UPDATE
already treat a key_find miss as a benign skip (bug-1970, which
explicitly anticipates post-purge facts); persist_verify_apply_correctness
counts only within the boot replay window. The TRANSIENT TTL now takes
live effect too (previously checkpoint-exclusion only).
UTs: kvmgr del INT+STRING (miss propagation, sibling intact, freed-slot
reuse, killer invocation), controller test_job_live_gc.c (expired DONE
purged + refcount 0, fresh/RUN retained, idempotent re-run, EXIT/PERR,
TRANSIENT TTL + transient_since==0 guard, standby no-op, NULL-safe,
arming guard). xcelib 761/761, controller 1354/1354.
E2E (t1): promote arms the sweep; the pre-restart DONE array jobs were
purged across five sweeps (33 jobs) as their 3600s windows elapsed;
submit→RUN→END flows and all daemons healthy throughout.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.
1. YAML resourcemap wiring (admin-plane-gaps 1.10):
- schema: Resource.distributions @8 / hostBased @9 uncommented --
resourcemap had no wire destination at all before this.
- resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
(distributions + hbased_defined); resobj_destroy frees them. Deep
copy is mandatory: the source is myconf->cluster_config, which
keeps owning and later frees its tree.
- controller: reconcile_config_resources + seed_sharedinst_from_config
(cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
distribution carries a value (the license-pool shape), from both
config_activate_committed paths, ahead of the conf4sched push.
- xceadmin: mapping.py maps resourcemap -> distributions (only the
location:[all]+value shape sets SHARED|GLOBAL), build.py renders
shared=true/distributions; golden expected.bin regenerated.
2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
seeded push freed the resval entries AND the backing array via
free_name_val_list, then xlist_fini freed the same array again.
Detach xreslist before the FREEUP/xlist_fini block (same ownership
class bug-1506 fixed in sched_worker_change). Latent since inception:
every pre-1.10 push ran with nresvals==0 and never armed it.
3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
queue_init (create) MOVED the definition's users/hosts/thresholds/
window payloads out, and queue_refresh_def MOVED them out again --
but reconcile_config_queues runs MORE THAN ONCE over the same
cluster_config object (boot: pre-election activation AND phase2's
build_queues), so the second pass refreshed live queues from the
drained definition: thresholds became the poisoned (count>0,
array=NULL) pair, the generated encode_Threshold_list has no NULL
guard, and the next queue/worker push died on the NULL base.
Both paths now deep-copy (queue_init and queue_refresh_def); the
definition keeps its payloads, (NULL/0) genuinely means "no list in
this config", and reconcile is idempotent. New dup_group_config
helper in utils.c (the generated code has no GroupConfig dup).
Mirrors resobj_init's distributions deep-copy and worker.c
apply_join_state's dup pattern.
4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
reconcile of a threshold-carrying queue and asserts definition
intactness plus a real encode_QueueConf -> wire -> decode round
trip; new test_config_resource_reconcile.c covers seed/deep-copy/
non-seed classes and idempotent re-seed.
Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three fixes so a pushed config edit takes effect on the RUNNING cluster
instead of waiting for a scheduler rejoin (admin-plane-gaps 1.1):
1. queue_refresh_def (controller/queue.c): reconcile_config_queues no
longer leaves existing runtime queues untouched -- it refreshes their
CONFIG-owned fields in place (resreq/pre_exec/post_exec interned
swaps; users/hosts/thresholds/windows via free_* + MOVE_PTR, same
ownership as queue_init; scalars copied) and notifies the live
scheduler/apiserver via the same NO-WAL hook a fresh queue gets.
2. config_refresh (controller/cluster.c): cluster_build_config is now
idempotent -- boot creates the confvar entries, activation re-runs
refresh their interned values in place (makeConfig refuses EEXIST,
kvmgr has no del_key), so heartbeat_interval/
inactive_heartbeat_count exports stop being frozen at boot values.
3. activation re-push (controller/config_repl.c): config_activate_
committed now calls cluster_build_config then xcesched_push_configs
(un-static'd) to the connected scheduler, on both the hash-verified
and -ENOENT fallback activation paths. Lock order follows the
established apply_pump_lock -> sched.lock seam; push failure only
WARNs (scheduler keeps its previous config until next join/push).
E2E-found display gap: apiserver's Go handleNotifyData had no
NotifyData_data_schedUpdateQueue case, so the controller's per-queue
notify was silently dropped and `xce queues list` kept the join-time
snapshot. Added the case (OnQueueUpdated) so hot-applied priority and
queue state changes are visible immediately.
Unit tests: reconcile refresh-in-place + notify counts (test_config_
queue_reconcile.c); activation repushes once per activation, not while
deferred (test_config_repl.c capturing fake); OnQueueUpdated in-place
refresh (handler_test.go). Full controller suite green.
E2E on t1/t2/t3: two config pushes (gen 8->9->10) with no restarts --
controller activation + per-queue refresh + scheduler re-push logged,
scheduler applied at the round boundary, `xce queues list` showed the
new priority immediately, and a submit->RUN->DONE job round-tripped on
the hot-edited queue with all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
P1-1: sched_resolve_job_picker() now validates all nine MUST hooks --
the four state-change hooks (on_queue_update/on_job_push/
on_job_state_change/on_job_update) were called unconditionally by
core_events.c, scheduler.c and snapshot_load.c but never validated at
resolution time, so a picker omitting one segfaulted on its first event
instead of failing startup validation with -EINVAL. Doc comment on
core.h's declaration updated to match.
P1-2: handle the picker hooks' return values instead of dropping them.
The known real producer of a rejection is builtin_fcfs's "queue not
(yet) in the candidate registry" -EINVAL: the job stayed tracked in
core->jobs but was never a scheduling candidate, with no log at all --
invisible forever (no dispatch, no PEND reason, nothing). Five sites in
core_events.c now WARN on rc < 0: the JOBS_PUSH fresh-job site plus a
root-cause WARN in core_upsert_queue (a rejected queue makes every job
in it fail too), both JOBS_PUSH resync-merge on_job_state_change sites,
JOB_STATE_CHANGE and JOB_UPDATE. All non-fatal: the push itself still
succeeds and the job stays tracked.
P1-3: sim_bootstrap_picker() is now name-driven with an explicit WARN
and graceful degrade for any picker name that has no sim-viable builtin
picker -- including the legacy "xce_sched_fcfs", whose job_elect_fn
reads plugin->sched (an xce_scheduler_t* the sim driver never has, and
which plugin_register_core() deliberately leaves NULL with no legacy
wrapper), so bootstrapping it would crash the rebuild walk on
picker_iface == NULL. Previously an unknown name silently returned
while sim still warned downstream. A short experiment with bootstrapping
the legacy plugin via an fcfs_new_plugin() factory was reverted: making
that plugin core-only-viable is a refactor of a plugin slated for
removal (audit P2 #18), not a bootstrap fix.
Tests:
- sched_picker_wiring_test: new test_picker_missing_state_change_hooks_
fails_with_einval wires the five ORIGINAL MUST hooks and leaves ONLY
the four state-change hooks NULL, so the asserted -EINVAL can only
come from the new checks.
- sched_core_events_test: new test_jobs_push_picker_rejection_still_
tracks_job_and_succeeds proves the push succeeds, the hook fires
once, and sched_get_job() still resolves the job after a rejection.
P2 items from the same audit are recorded (not fixed) in gaps.md
section 七 (items 18-21): legacy fcfs zombie, dead scaffolding fields,
preemption without a producer, job-priority chain.
134/134 tests pass.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Limit-subsystem audit fixes (#1, #2, #4b, #5, #7; #3/#6/#8 deferred):
#1 PEND reason semantics inversion (the real fix):
- plugins/limit.c: remove the bogus rc<=0 PREASON_LIMIT_GLOBAL/HOST attach
blocks from match_job_global_limit_()/match_job_host_limit_(). match_limit()
only knows scope membership (1 = in scope, 0 = not), never usage-vs-cap --
the attach fired "quota exhausted" for every job OUTSIDE the limit's scope
and never for a real exhaustion.
- resource.c sched_try_alloc_on_host_() + alloc_runner.c
sched_alloc_prepare_shared_(): attach the real verdict in the generic
satisfy check's unmet loops -- an unmet resource with XCE_RESF_LIMIT now
reports PREASON_LIMIT_HOST (per-host loop, host named) / PREASON_LIMIT_GLOBAL
(shared loop, host-less) instead of the generic NO_RESOURCE; the vres name
("limit=<name>[/u=..][/q=..][/p=..]") rides in the reason's resource field,
reverse-parseable to the bare limit name.
#2: rtr_reason_name() now covers codes 4-9 (aligned with pendReasonName() in
the apiserver and reasonScenario() in the MCP server); preason.h's
main-reason comment states the real producer contract.
#4b: make_sched_obj() normalizes obj->project ""/NULL -> "default", matching
sched_get_project()'s default -- a limit scoped to projects.members:[default]
now matches obj->project the same as job->project (the wire decodes an absent
project as "", not NULL).
#5: percent_capacity_resource_map: "swap" -> "swp" (the live load index name
workers register; "swap" matched no live resource, so a percent limit on it
fell back to the live-value base bug-1834 exists to avoid) + add tmp ->
maxtmp. Fixture and lib tests follow the live names.
#7 (coverage): test_alloc_runner.c asserts an unmet shared limit vres gets
PREASON_LIMIT_GLOBAL; test_alloc_runner_search.c asserts an unmet per-host
limit vres gets PREASON_LIMIT_HOST with the host named;
test_repl_trace_e2e.c / test_trace_render.c and the repl_trace_baseline
fixtures (XCE_REPL_BASELINE_REGEN=1) follow the corrected reason rendering.
Full tree: 134/134 tests green (scheduler 100, libxce, harness, controller,
worker, autoscaler, apiserver).
Co-Authored-By: Claude Code <noreply@anthropic.com>
A failed test run's evidence (meson-logs/testlog.txt + the mkdtemp sweep's
FAILED-case /tmp/xce_* fixture dirs) was destroyed by the next build:
setup_build rm -rf's the build dir, and any subsequent meson test overwrites
testlog.* in place. t2 2026-10-22: a controller case failed at ~05:46, the
make clean + rerun at ~05:52 left 134/134 OK and zero trace of what failed.
Two chokes in xce-build.sh:
- finalize_failure_evidence (EXIT trap): on rc!=0 with RUN_TESTS=1, copy
meson-logs to testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/ and drop
the .meson_test_failed_rc marker; also releases the build lock (the old
lock-release trap folded in).
- archive_previous_failure (run start, pre-wipe): if the marker exists,
move the previous run's meson-logs to prev-<stamp>-rc<N>/ -- outside the
build dir, unreachable by the wipe; sweep aged (>5 min) /tmp/xce_*
FAILED-case fixture dirs into prev-<stamp>-fixtures/ so they sit with the
testlog that explains them; cap the archive at newest 10 per build dir.
Also -type d on BOTH /tmp/xce_* sweeps (this one and bug-924's
prune_stale_tmp_fixtures): mkdtemp fixtures are always directories, but
the bare name also matches regular files -- the untyped find live-moved
daemon debug logs (xce_scheduler.log, xce_strace*.log) into the archive,
and the bug-924 variant rm -rf's its matches.
Test: src/tools/test-failure-evidence-archive.sh, 6 scenarios / 11
assertions exercising the real function bodies (sourced with surgical
stubbing) -- all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two independent fixes:
1. Worker pre-exec failure (TODO fix). queue_pre_exec()/job_pre_exec()
previously called exit(1) on failure, killing the one-shot worker
before any status report reached the master -- the job then hung with
no user-visible reason and the worker process vanished. Now the
failure path reports the job FAILED via ipc_worker_send_status with
stage JOB_EXEC_STAGE_{Q_PRE,PRE}_EXEC and the command's real exit
code, skips the job body (no post-exec -- the body never ran,
mirroring LSF pre-exec semantics), and takes the same tail as the
VNC/X11 start failures. main.c now carries job_status->stage through
to xcecd_job_finished() so the stage (not the MAIN default) reaches
the controller's fail fact, deciding both the exit-status merge
(apply_job_fail_transition) and whether the exit code is recorded
(apply_job_fail_exit_code). Also fixes the error print: uv_proc_pre_exec
returns a WEXITSTATUS, not a uv errno, so uv_strerror mangled a plain
exit status into "Unknown system error N".
E2E verified on t1: job-level pre-exec `exit 7` -> job EXIT,
worker logs "Job pre-exec command failed with exit code 7", marker
file shows only the pre-exec ran (no post-exec, no job body), and
the exit code reaches the apiserver's OnJobFinished (exit_code=7)
and the committed WAL_JOB_FAIL fact.
2. Snapshot queue no longer in config wedged election forever.
Checkpoint snapshots record every queue in the live qmgr; if a queue
is deleted from queues.yaml between the checkpoint and a restart,
restore_queue_state returns -ENOENT for it ("queue not yet built
from config -- skip" per its own contract), but
load_snapshot_entities treated any non-zero as fatal ->
persist_recover failed -> cluster_init_phase2 retried forever
(bug-1477 never self-terminates) and no primary was ever elected.
The snapshot queue loop now treats -ENOENT/-EINVAL as a benign skip
with a WARN, matching the same documented policy as
apply_committed_entry's and apply_wal_entry's WAL_QUEUE_CTRL cases;
every other non-zero return stays a genuine failure. Doc comments
updated accordingly.
Live-reproduced and verified on t1 (2026-10-22): deleting a test
queue + restarting the controller previously wedged election
permanently; with the fix the controller boots past the stale frame
with a WARN and reaches CTRL_PRIMARY.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Controller suite flake (user-reported, -j parallel runs): the commit
engine's async timeout sweep armed continuation deadlines with the
hardcoded DEFAULT_SYNC_TIMEOUT_MS (500ms) instead of the configured
sync_timeout_ms, while the blocking persist_commit_barrier honors the
config. Fixtures configuring longer or shorter sync timeouts saw the
sweep resolve -ETIMEDOUT at 500ms before their explicit
truncation/second-call steps, flipping "Expected -17/-2 Was 0/-110"
assertions (test_commit_resolve.c:1458/:1616). commit_submit_async and
commit_sweep_worker now both use p->config.sync_timeout_ms with
DEFAULT_SYNC_TIMEOUT_MS only as the zeroed-config fallback (mirroring
persist_commit_barrier's timeout_ms == 0 convention), and the sweep
WARN reports the effective value.
Apiserver flake (found during full-build verification):
deriveArrayJobStats took the array identity from the first element
seen over store.Jobs.List(), but List() walks a Go map whose order is
randomized per call -- when element rows disagree (the pinned test
case), the stats identity flipped run to run (~12% of -count=1000
runs). The identity carrier is now the lowest Unique_JobID (the array
head, element 1), tracked via arrayAgg.carrierUID; other rows
contribute counters only. Counters were already order-independent and
are unchanged. The test now pins the head-carries-identity contract
explicitly.
Also fixes a latent parse-error trap in the test: a string literal
ending a line (C-style implicit concatenation) triggers Go's automatic
semicolon insertion -- the next line's argument list parses as
"missing ',' before newline". String concatenation across lines must
keep the '+' at the end of the line.
Verified: controller suite 1342/0 (three runs incl. -j 7 parallel
stress); sweep WARNs now match fixture configs (300ms fixture warns at
300, longer-config fixtures emit none); go test -run
TestDeriveArrayJobStats -count=1000 all green (was 118-130 failures);
full ./tools/xce-build.sh all green end to end.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The single JOBS value was doing double duty: the PROCESS_FACTOR
forked-children discount baked into it made sense for meson test -j and
go test -p (flat process caps with no live check), but meson compile
consumes the same number as ninja's -l LOAD-AVERAGE ceiling -- and -l
already re-checks the live run-queue before every launch, so applying the
halved number there too just capped compile parallelism twice. On t1 the
recalibrated 512MB budget produced mem_jobs=9 but cpu_jobs=(8-0.4)*0.5=3,
leaving the memory recalibration without effect on the compile step.
Now: COMPILE_JOBS (raw nproc-minus-load, memory-capped) feeds -l; JOBS
(the same value discounted by PROCESS_FACTOR) feeds the flat test caps.
An explicit --jobs N still sets both, N and N, as before.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The original 1536MB/job figure was calibrated around two memory hogs that
the build/de-nix work has since removed: the 293MB-peak serial capnp
compile (now compiled once via libxce_dep) and the 17 scheduler test
executables each statically linking all of scheduler_srcs (now one shared
sched_srcs_lib archive; link peaks ~125-150MB with -g1). Measured per-job
compile RSS ceiling post-dedup is ~125-150MB; 512MB keeps ~3x headroom
while letting an 8GB box run ~8 concurrent compiles instead of 2-3
(t1: mem_jobs goes from ~2-3 to ~9, so the -l ceiling becomes CPU-bound
rather than memory-bound).
SKILL.md's hand-computed formula is updated to match.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Production (hand-fixed, semantics preserved with width-limited %.*s):
- persist.c (33 sites): workdir/name joins share the destination budget;
WAL segment/snapshot paths via a WAL_SEG_WALPATH_MAX macro derived from
seg_path's own declared size; trim loops use d_name's NAME_MAX (255) cap.
- config_repl.c (16 sites) and ha.c's config-hash twin: same half-split
and fixed-literal-tail budgets, byte-identical path construction kept
between ha.c and conf_save_config.
- main.c: certdir/key/cert paths leave room for their fixed suffixes.
Test suites (-Wno-format-truncation, rationale in-tree): the 68 remaining
sites are all snprintf(dir, 4096, "%s/<literal>") fixture scratch paths
under mkdtemp /tmp/xce_* dirs that never approach PATH_MAX -- gcc cannot
prove the tail fits, but every one is a false positive. Applied as a
compile_args declare_dependency on each suite's shared deps list.
Vendored c-capnproto: -Wno-switch (~40 exhaustive-dispatch false
positives), -Wno-unused-function (empty define_encode_function stub),
-Wno-format-overflow (255-byte uniontag + 3-byte prefix vs 256-byte buf,
unreachable schema lengths). Both the wrap checkout and the
packagefiles overlay patched identically.
Also: test_worker_launch_type_dispatch removes its fixed-path fake-xrun
/tmp fixture at exit (bug-924 /tmp hygiene; the mkdtemp shim can't see it).
Verified: fresh full recompile of every source = 0 warnings (gcc 13.3,
-Wall asan profile); meson test 134/134 green; no /tmp/xce_* residue.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The sched_unit_test aggregate and 16 zero-#include standalone tests each
compiled all ~50 scheduler sources as their own TUs (~800 redundant
compiles of identical sources per clean build). Add a shared
sched_srcs_lib static archive (scheduler_srcs minus main.c) in the parent
meson.build; those 17 targets now link it and compile only their own test
TUs.
- Only extern-API consumers convert: these tests call nothing but symbols
declared in scheduler.h/core.h. Textual includers (white-box statics
access) keep their own compilation -- their TUs define the included
files' symbols and could collide with archive members; the 4
exclusion-list foreach blocks (snapshot_load_allocator_regression,
repl_trace_wiring, trace_error_paths, trace_bounds) are untouched.
- -Wl,--wrap= interposition is link-time symbol redirection and works
identically against archive members; the three --wrap tests
(publish_retry/backpressure/redelivery) are the sensitive probes and
pass.
- The aggregate keeps -Werror=incompatible-pointer-types and all baked
path defines; its src foreach is gone (the lib carries the sources).
Verified: fresh asan tree 1135 steps (was 1934), ninja green, meson test
134/134 including all converted targets; disk unchanged at ~1.3G after
the previous commit's capnp dedup.
Co-Authored-By: Claude Code <noreply@anthropic.com>
libxce_dep's sources:[gen_srcs] handed every consumer target the generated
.c output, so 143 targets each compiled controller.capnp.c independently:
a 9.5s / 293MB-RSS / 3.3MB-object TU under the asan profile, ~35% of a
clean build's serial work and ~470MB of duplicate objects on disk.
Index the dep's sources to gen_srcs[1] (the .h output) instead: the
custom_target edge still orders every consumer's compiles after codegen
(bug-1798's guarantee, unchanged), .h sources are never compiled, and the
.c is compiled exactly once inside libxce. libxce_src keeps the unindexed
custom_target for the same reason it always did.
Verified: fresh asan tree builds green (1934 steps vs 2076), meson test
134/134 including the three --wrap interposition tests; tree size 2.0G ->
1.3G.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Collapse the 28 zero-#include white-box scheduler tests (whose meson
executable blocks each duplicated the full scheduler_srcs list) into a
single sched_unit_test aggregate, following the controller
controller_test.c precedent: one executable, one process, one runner.
- sched_unit_runner.c: SUITES registry (28 entries) with per-suite
setUp/tearDown dispatch, -t <suite> filter, UnitySetTestFile per entry
- testsuite/meson.build: runner + 28 test TUs + every scheduler_srcs
entry except src/main.c as separate TUs; foreach over scheduler_srcs
so the aggregate can never drift from the production source list;
all baked-path defines (XCE_SCHED_*_PATH, XCE_SIM_*) move to the
aggregate's c_args; -Werror=incompatible-pointer-types carried
suite-wide (verified clean across all 28 tests + 50 src TUs)
- transformed tests drop main()/setUp()/tearDown() and local stubs that
now resolve against the aggregate's real src TUs
- test() registrations 126 -> 99 (126 - 28 + 1 aggregate); the three
--wrap= interposition tests (decision publish retry, backpressure,
decision redelivery) keep their standalone executables unchanged
- isolation libs (sched_core_isolation_lib, sched_sim_isolation_lib)
stay build-only and are wired as test()-level depends; the nm(1)
text-scan tests find them via XCE_CORE_LIB_PATH/XCE_SIM_LIB_PATH
Verified: meson test --suite scheduler 100/100 green x3 consecutive
runs (43s total, aggregate 3.3s); /tmp/xce_* sweep clean (the one
leftover is test_worker_launch_type_dispatch.c's by-design fixed-path
fake xrun launcher).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Deduped census insight: the earlier "730 warnings" figure counted per-TU
echoes of the same source line (white-box textual inclusion recompiles
scheduler.c/limit.c/etc. dozens of times, each copy re-reporting). The
distinct warning sites in our code were 24 unused-variable + 11
unused-function + 2 Wcomment. This commit removes all of them; a clean
full rebuild now reports zero warnings outside the format-truncation
family (125 sites, deferred to its own pass) and vendored capnpc-c.
Notable non-mechanical fixes (the rest is dead-code deletion):
- schedobj.c make_sched_obj() silently swallowed its sub-constructor's
rc (ENOMEM/-EINVAL) and returned 1 unconditionally -- a half-built
schedobj with a NULL union payload could escape to callers. It now
propagates the failure, and make_sched_job() frees and returns NULL
on it instead of continuing with a broken schedobj.
- scheduler.c make_allocs(): vget() rc was set-but-unused; the r==NULL
skip below is the intended handling, so the call is now (void)'ed with
a comment saying so.
- commit.c highest_appended_lsn(): kept but scoped #ifdef XCE_TESTING --
it backs the never-beyond-the-log assert; it had no production caller.
- limit.c get_vres_value(): deleted -- its only call site sits inside a
#if 0 block and its own body was #if 0'ed to a constant.
- group.c update_group_dirty(): deleted recursive dead subtree, no
callers outside itself.
- test_sim_reconcile_unaccounted.c: test_reconcile_respects_host_
capacity_under_contention() was never registered in RUN_TEST; removed
with an NB pointing at the registered bug-1836 coverage.
Verified: clean full rebuild = 0 unused/Wcomment/implicit warnings in
our code (89 unique sites remain, all -Wformat-truncation or vendored);
controller_unit_test 1342/1342, xcelib_test 758/758, 126 scheduler
suites and 18 worker suites all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Warning fixes (from a 730-warning full-build census; these were the
semantically risky ones):
- test_repl_majority.c:272: remove a broken "field-existence" loop whose
condition (i < ep != NULL) parsed wrong (-Wparentheses); the direct
pending_ack_count write it guarded already exists and still fails the
build if the field disappears.
- ha.c:8593: ha_rdhup_election_hook now takes const xce_candidate_t**
so RDHUP election can hand over candidates straight from the const
peers_cfg without stripping const. All 5 test-side hook impls only
(void)peers -- signature updated in ha.h + 3 test files.
- ha.c:726/872/10804 endpoint_connect: constify the host parameter of
endpoint_connect/socket_connect (netio.h/netio.c). The functions only
pass host to gethostbyname(); peer->hostname etc. can now flow in
without discarding const. All callers are read-only and unaffected.
Debug profile (src/meson.build): drop -gdwarf-4 -g3
-fvar-tracking(-assignments) in favor of -ggdb -g1. Debug objects were
~16x larger with full DWARF (scheduler.c: 1.3MB vs 0.08MB), pushing the
build tree to 3.9G; at -g1 it is 850M and a clean build drops from
6m43s to 4m36s. -g1 keeps file/line and backtraces; gdb variable/
macro inspection needs the asan profile. -fno-omit-frame-pointer kept.
Verified: controller_unit_test 1342/1342, xcelib_test 758/758, all 126
scheduler suites + 18 worker suites green; modified sources compile
with zero warnings.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Section 3 documented the retired nix path (nix.conf edits, nix build
.#xcecore, AppImage, nix develop self-wrap). Replace it with the actual
build as it now works on plain Ubuntu:
- 3.1 one-shot apt install (toolchain + 11 C libraries), pointing at
DEPS.md for the versioned dependency table and the wrap-subproject
source closure for everything apt does not carry
- 3.3 daily development via tools/xce-build.sh (profiles, --incremental
caveat), plus the plain meson commands it wraps; Go vendor refresh
without any special shell
- 3.4 tarball packaging via tools/xce-package.sh + install.sh (the
shipping distribution form, replacing the AppImage section)
- 3.5 scripts/ci-ubuntu.sh as the CI entry point
Co-Authored-By: Claude Code <noreply@anthropic.com>
ttyd is the interactive-job launcher for worker TTY/DISPLAY modes: the
worker resolves it next to its own binary (inter_job_utils.c
readlink(/proc/self/exe) + /ttyd, /opt/xce/bin/ttyd fallback), so a
tarball without bin/ttyd silently loses -I/--tty on the target host.
The packager now picks it up from $BUILD_DIR/worker/ttyd first, then
/opt/xce/bin/ttyd, and warns loudly if neither exists.
NB it is NOT an upstream release: this is rshen's modified ttyd (version
string 1.7.7-eccebc6), and the binary we bundle today is that same
modified build. Source lands in-tree later and moves to a meson target;
the packager already prefers a build-dir copy for that day.
DEPS.md: document the ttyd situation (including the MIT-compatibility
audit needed before commercial distribution) and record the user-flagged
license note on bundling libhwloc/libedit/libtirpc -- fine for the
development/test rollout, needs a per-item license audit (bundled
LICENSE file at minimum) before commercial release.
Co-Authored-By: Claude Code <noreply@anthropic.com>
A committed job-state fact (WAL_JOB_UPDATE/FAIL/START/...) whose jobid no
longer exists in the jobmgr used to return -ENOENT from
apply_committed_entry(). Both of its callers (replay_wal during boot,
persist_apply_pump_run live) halt-and-fence on rc < 0, engaging the Live
Apply Resource Fence. A finished job is purged from the live table once
finished_retain_s expires, but its committed WAL entries remain; a boot
replay walks the full log and re-hits a late post-finish UPDATE for the
purged jobid on EVERY restart. No in-memory state change can ever make a
purged job re-appear, so the fence can never self-heal: the controller
logs "promotion FENCED -- durable_floor < cluster HWM=0" every 30s
forever, rejects all component joins ("standby has no primary yet"), and
cannot serve. Observed live on the t1 rollout (2026-10-21): the old
primary had logged the same entry as "duplicate UPDATE to state 0x4 --
no-op" while the job was still in memory; the first restart after the
job's purge tripped the fence. Pre-existing since the July 2026
apply_committed_entry dispatch table (2fa955b9/387ea35c) -- exposed by
any controller restart, not a de-nix regression.
Fix: all three case groups (shared START/FINISH/STOP/RESUME/RUSAGE/CKPT/
PREASON/DISPATCH group, WAL_JOB_FAIL, WAL_JOB_UPDATE) now return 0 for a
missing job, mirroring apply_wal_entry()'s existing behavior on the same
seam (restore_job_state()'s rc discarded; the FAIL branch guards on
j != NULL) and this function's own WAL_QUEUE_CTRL benign-skip precedent.
Genuine construction/ENOMEM failures keep their fence semantics.
Also:
- test_apply_committed_entry: three new tests covering the skip for all
three groups (incl. the POSTDONE-mark variant), asserting rc == 0 and
no hook fires.
- lib/meson.build: ASAN_OPTIONS=detect_leaks=0 on the capnp generation
command. capnpc-c deliberately never frees its compile-time structures
(one-shot tool); under the de-nix asan profile the wrap-built generator
carries -fsanitize=address and LSAN's end-of-process check failed the
generation step. The legacy nix build never propagated -fsanitize to
the subproject. Same exemption convention as every other one-shot/
never-torn-down boundary in this repo.
- xce-package.sh: ALWAYS-bundle name list (libcyaml/libyaml/libtirpc/
libcgroup/libhwloc/libedit) alongside the ldd-location rule -- libcyaml
and libcgroup proved missing on the t2/t3 Ubuntu 24.04 runtime hosts
even though the build host resolves them from /usr/lib.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The build no longer self-wraps into `nix develop`: src/tools/xce-build.sh now
runs meson+ninja directly on system packages, with a dependency preflight
that fails early naming the missing tool/pkg-config package (DEPS.md).
Dependencies (see DEPS.md for the full tables):
- 11 C libraries from apt (libuv, openssl, libedit, sqlite3, hwloc, libcgroup,
libtirpc, libcyaml, libyaml, zlib, libsystemd) with version floors in
meson.build mirroring the Ubuntu 24.04 baseline.
- sd-bus: default libsystemd, -Duse_elogind=true opt-back for nix-era hosts.
- 6 libraries/tools unavailable in apt become committed meson wrap
subprojects under src/subprojects/ (source closure, ~500 files): the
upstream trees plus packagefiles/ overlays and diff_files patches
registering dependency/program overrides the nix .pc files used to
provide (incl. the '<name>-raw' variable deps the Go CGO path reads).
Fresh-clone reproducibility verified: wiped trees + `meson subprojects
download` reproduce the committed state exactly (fixed librtti.wrap's
dangling patch_directory and c-capnproto's empty packagefiles overlay).
- xceadmin: self-contained venv built from requirements-build.txt (pinned,
offline-seedable via PIP_FIND_LINKS) instead of --system-site-packages.
Build/packaging:
- Makefile facade over xce-build.sh; scripts/ci-ubuntu.sh drives a bare
Ubuntu 24.04 container through deps + build + tests.
- src/tools/xce-package.sh: release-tree tarball packager. Collects
non-base DT_NEEDED libs from the BUILD-tree binaries (staged ldd lies:
the dangling build RPATH makes ldd drop the very lines the naive filter
needs), sets $ORIGIN RPATHs, rebases the xceadmin venv at install time
(kernel does not parse quotes in shebangs), ships systemd unit samples +
install.sh. Local smoke test: all 8 binaries resolve, xceadmin works from
both bin/ wrapper and venv script.
Test fixes found on the way (all environmental, not build bugs):
- harness_test.c setUp(): pin XCE_CONTROLLER_PEERS to a bound-then-closed
loopback port so the CLI-error tests assert the no-controller failure
path even on a dev box running a live cluster on localhost:9990.
- test_test_cli.c: setpgid(0,0) in the spawned-xce_test child. libedit's
sig_handler re-raises received signals via kill(0, sig) -- process GROUP
wide (strace -k evidence) -- and the child previously shared the
harness's pgrp, so an interrupted SSL handshake killed the whole suite
(exit 130).
Also fixed while re-baselining on the de-nix toolchain (pre-existing, now
deterministic): raft_init zeroes pre-vote tally state (pre_vote_in_progress/
pre_votes_granted/pre_vote_replies were left at stack garbage);
push_config_to_standbys clears stale config acks before fan-out
(repl_reset_config_acks) -- SHA-256-derived tags are not monotonic, so a
previous push's ack numerically above the new tag fabricated a barrier
quorum; capnpc-c's generated encode_* memsets the destination first
(skipped Data fields fed garbage into capn_setp).
Validation: asan profile 161/161 meson tests green (one known p2_integration
-110 timing flake on first run, green on re-run); debug + release full
builds OK; tarball smoke-tested on the build host. Cluster rollout on
t1/t2/t3 follows.
Co-Authored-By: Claude Code <noreply@anthropic.com>
