Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).
Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.
Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
queue refuses new submissions only; pending jobs still run) --
supersedes our QUEUE_CLOSED preason on the dispatch path;
sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
rounds (sched_stash_config/sched_apply_pending_config), applied at
the same main-thread round boundary that preserves bug-1513
serialization
- bug-1760: sched_check_pending_rejoin exposed +
xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
(superset of our bug-1500 execuser fallback; our job-typed
resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
src/harness/test.c; controller suite split into
controller_test.c + fixtures; their clitest fixtures spawn xce_test
mid-suite under `meson test -j`
Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
gate re-ported into their relocated allocator (sched_election.c)
after plugin_job_election, with attach on the allocator thread
only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
merge initially re-carried our first-detail-add-promotes-main_reason
fallback in add_job_preason (4e2ac86), but their whole trace/verify
stack treats "detail-only => main_reason 0" as a contract --
sched_trace_capture counts main_reason as its own occurrence
("Expected 2 Was 3" in test_trace_capture), the outcome renderer
prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
and the committed .replay/.sim_verify baselines encode main_reason=0
(test_repl_trace_baseline byte compare, test_trace_equivalence
off-vs-traced). The fallback is dropped from the shared lib;
the user-visible "overall reason" survives as an API-layer view:
rawJobToModel (convert_pure.go) promotes the first detail row to
main_reason_name when main_reason is 0 (main_reason VALUE marshals
unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
(their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
resend, xcecli_force_reconnect + their is_connection_broken both
exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
/xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)
bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
(testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
harness_test.c -- hundreds of cases per process) call
case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
dirs are swept immediately in tearDown (after engine reaping), FAILED
cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
disables. One full controller_unit_test run was observed to leak 12G+
to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
failures
- harness_unit_test_deps links the shim library as well
Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
detail-keeps-main-0 contract; new tests cover limit-code detail
rows and explicit-main-wins-over-later-details; apiserver
convert_test.go gains the detail-only name-promotion cases
Co-Authored-By: Claude Code <noreply@anthropic.com>
The control API already takes `signal` (kill only: that signal once,
instead of the graceful INT, TERM, KILL sequence) and `force_kill`
(SIGKILL at once); xcecli could not send either. `jobs kill` now has
-s/--signal <name|number> (names in any case, SIG prefix optional, sent
upper case without it) and --force, in both the job-ID and the filter
request; using both at once, or an empty signal, is refused before any
request. stop and resume get neither (the server rejects a signal there).
Kill help and docs/xce.md "Killing and stopping jobs" document the flags.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- src/docs/xce.md: explicit signal and force kill are REST-only (signal,
force_kill); xcecli has no signal option yet.
- src/docs/deployment.md: app.yaml row lists the job block
(max_array_size, terminate_interval) and trusted proxies.
- xcecli jobs kill/stop/resume help describes the signal sequences.
- REST: control endpoint annotations and the signal/force_kill model fields
describe the behaviour (swagger output not regenerated: swag fails on an
unrelated annotation).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The loader now stores the effective interval (10 s when unset or 0, LSF's
JOB_TERMINATE_INTERVAL default) and refuses a negative one; app_schema.json
describes the setting; src/docs/xce.md documents kill/stop behaviour and
the interval.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
deployment.md: auth.yaml now holds bcrypt hashes of the (still public)
dev credentials with startup warnings; JWT secret from XCE_JWT_SECRET or a
random per-process secret; logout revocation is per process;
security.trusted_proxies for reverse proxies.
security.md steering: JWT parsing, revocation, secret resolution,
trusted proxies, middleware abort rule, bcrypt and role-change rule.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- apiserver peek (parseJobIDParam): <array-id>[k] now resolves to the
element; it used to drop [k] and look up the array identity itself
(and panicked in parseArrayName on some inputs). A job number still
names that job exactly (an element by its own XCE_JOBID); ranges and
invalid selections are rejected.
- apiserver restart: the decimal-only pre-check (isValidJobId, now
unused and removed) rejected the written form c-a-ts-seq[k];
ParseIdentitySelection, already used for the lookup, validates it.
- xcecli jobs restart converts its argument with ToAPIJobID like the
control commands, so written IDs work and invalid ones fail locally.
- docs: the peek/restart caveat in the Job arrays section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
src/docs/xce.md gains a 'Job arrays' section: name[1-N] and name[1],
numbering from 1, the API-server limit (default 1000, at most 32767),
<array-id>[k] and <array-id>[a-b] selection (with the peek/restart
caveat), XCE_JOBINDEX, the harness jobsub -a N / $jobid[k], and every
LSF array feature this release does not support. The superseded
statements in job-array-single-wal-record and xce-test-e2e-primitives
are annotated (history kept), the controller-job-cleanup and
job-dependency briefs describe 1-based numbering, and the roadmap gains
an entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.
Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
The apiserver's host max_mem is already in bytes (the worker reads
sysinfo().totalram × mem_unit → bytes and passes it through unchanged via
controller.capnp), but ListNodes treated it as MB and multiplied by
mbToBytes, inflating each node's reported memory to ~8.2e15 (~7.3 PiB).
Now reports the raw bytes (~7.7 GiB per node).
- httpclient.go: drop ×mbToBytes for node max_mem; clarify mbToBytes is
job-submit-path only (job max_mem is never populated → stays 0)
- ordermap_test.go: correct assertion that had encoded the buggy value
- httpclient_test.go: regression test TestListNodeMemoryBytesKeepsByteUnit
- buglog.json: bug-1514 + close-tags for the bug-1494 family (prior session)
Also add MCP demo deliverables:
- docs/claude-code-demo-guide.md: live Claude Code demo guide (7 scenes)
- docs/mcp-demo-design.md: leadership demo design
- docs/mcp-demo-run.py: add --slow/--color flags for screen recording
Co-Authored-By: Claude <noreply@anthropic.com>
- Change default record_dir from /var/log/xce/ to /tmp (always exists)
- Add mkdir(record_dir, 0755) at startup for custom --record-dir paths
- Split validation_error_text into 6 distinct per-errno messages
- Accept human-readable job IDs in 'info job' via parse_jobid()
- Add 'info jobs' command to list all jobs in sim REPL
- Format job IDs with fmt_jobid() for xce_test-consistent output
- Replace printf with fprintf(stdout, ...) in all REPL files
to bypass -Wl,--wrap=printf linker interception
The "Deploying the Scheduler" section predated scheduler-core and only
covered normal-mode startup. Adds session recording (--record-dir,
xceadmin scheduler record start|stop|status, filename validation),
xceadmin scheduler dump for offline .replay inspection, and simulation
mode (--sim, --sim-speed, --sim-verify, --sim-alt-events, --sim-output,
--interactive REPL) — all verified against the actual argp help output
and xceadmin cli.py source, not assumed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
worker/src/resource_sync.c init_builtin_load_indices registers 11 load
indices (r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io) but resources.yaml only
defined r1m/r15s/mem/tmp. The 7 missing ones (r15m/ut/pg/ls/it/swp/io) caused
controller "update_resval: Success" ERRORs and scheduler "unknown resource in
per-worker resvals update" warnings, leaving the worker resource view
incomplete. Add the 7 definitions matching the existing r1m/r15s pattern.
Docs: log bug-1494 (all jobs PEND, scheduler never dispatches; after this
config fix jobs still PEND — remaining suspect is the scheduler→controller
join path after restart), update design doc + verification manual known-issue
tables and root CLAUDE.md cluster-verification status.
Co-Authored-By: Claude <noreply@anthropic.com>
New §5 in src/docs/chatbot-verification.md: how to connect Claude Code to
the scheduler-agent MCP server. Covers building the mcp-server binary,
`claude mcp add` CLI registration (verified against installed CLI
v2.1.220), the .mcp.json project config schema, the three scopes
(local/user/project), the first-run approval gate, verification commands,
and stdio/env gotchas. Original §5-§7 renumbered to §6-§8.
Co-Authored-By: Claude <noreply@anthropic.com>
New src/docs/chatbot-verification.md: reproducible steps to deploy the
scheduler-agent stack on the t1/t2/t3 test cluster (AppImage build, store
sync, controller/apiserver/scheduler/worker startup) and verify Phase 0-1
(login, mcp_operator API key, read ops, submit). Documents the vcpus quirk,
setsid launch pattern, conf/current pointer mechanism, and known bugs
(job-control HTTP 500 = bug-1490, HA election deadlock).
Also add the manual to the design doc's related-documents index.
Co-Authored-By: Claude <noreply@anthropic.com>
trim_wal_segments + rotate_snapshots run on every checkpoint --
operators don't need to manually purge data/. Documents the
snap_keep_n=2 default and that it's a cluster-config field, not a CLI
flag.
--system-site-packages means pip skips vendoring click/pycapnp/pyyaml/
jsonschema since nix already provides them -- deliberate, to keep
pycapnp's ABI-sensitive build pinned to the exact capnproto C library
nix built it against. Activating the venv outside nix develop (or the
AppImage) hits ModuleNotFoundError immediately; not a packaging bug.
rshen/Letmein was a live, checked-in cluster_admin credential in
auth.yaml. Renamed to a non-identifying username with a clearly-marked
placeholder password.
docs: add full ops deployment guide (src/docs/deployment.md)
Covers building and running all four daemons plus xceadmin, based on
direct source verification (CLI flags, config file shapes, TLS posture)
and a live single-node deploy. Documents real production gaps (no
process supervision, plaintext non-controller traffic, exposed dev
creds) rather than treating the repo as a finished runbook.
- Mark Phase 1 as WIRE PROTOCOL COMPLETE (was: ACTIVE with stubs)
- Add commits 2ed7e34-5053c8d to component table
- Remove resolved limitations (do_one_config_fetch stub, push no-op)
- Update test count: 18 → 24 tests, total 84 passing
- Update Phase 2 table: handle_upgrade_config wired, push timeout enforced
- Update files changed to include request.c shims and new test additions
Replaces blocking pthread_join() per thread with a CLOCK_MONOTONIC
deadline + 5ms poll loop. CONF_PUSH_TIMEOUT_MS (500ms) now enforced.
Matches design §6.3 aggregate timeout guarantee.
- Add done field to push_ctx_t; push_thread sets it on exit
- Add static wait_all_with_timeout() before push_config_to_standbys()
- Wire push_config_to_standbys() to use wait_all_with_timeout()
- Add #include <stdatomic.h> to config_repl.c
- Guard push_ctx_t local typedef with #ifndef XCE_TESTING
- Expose push_ctx_t and wait_all_with_timeout_for_test() in header
under XCE_TESTING for unit test access
- Add test_wait_all_with_timeout_slow_threads (timeout fires, acked=0)
- Add test_wait_all_with_timeout_fast_threads (all complete, acked=3)
- Update design doc §6.3 with actual implementation signature and
5ms poll interval note
load_primary_config() reads from workdir/name/conf — conf_save_config()
was writing to workdir/conf. Latent bug: would have corrupted upgrade
restart cycle once Task 11 wired the upgrade path.
Update all conf_read_current_filename, conf_get_saved_tag, and
conf_check_stale calls in the test suite to use the corrected
effective_dir (workdir/name) path structure. Add new test
test_conf_save_config_path_consistency that explicitly verifies the
written path matches what load_primary_config() reads.
- myconf_t gains peers_cfg (xce_cluster_config_t) populated from --peers
- controller/main.c: parse --peers host:port,... via argp; ca_cert_path
auto-derived from workdir/certs/xce.crt (no --ca-cert flag needed)
- cluster_init: derive my_priority from position in peers list (index 0 =
primary); fall back to 0 when --peers not given (single-node mode)
P0 fixes (9):
- R-22: COW vstore find-slot-and-write not atomic
- R-23: vput grow path without mutex during COW
- R-24: Post-growth slot double-claimed by concurrent threads
- R-25: Missing key_snapshot field for checkpoint serialization
- R-26: malloc(0) for WAL_CHECKPOINT entries
- R-27: Fallback to older snapshot with trimmed WAL
- R-28: hdr.term > ha->term + 1 too restrictive
- R-29: _Atomic on enum type (implementation-defined)
- R-30: handle_jobsubmit() ignores hook return value
- R-31: malloc NULL check missing in standby receiver
P1 fixes (5):
- R-32: No heartbeat during persist_recover() on new primary
- R-33: Catch-up from dead primary removed from promotion
- R-34: Added total_wal_bytes field to persist_t
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
