merge: sync Rongsong's scheduler-framework (c5b172c7..3442a57d, 781 commits)
Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1, scheduler-agent Phase 2, bug-924 owner-pid shim). Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/ (92 files dropped); .claude/ + CLAUDE.md keep ours. Key theirs-side changes absorbed: - bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed queue refuses new submissions only; pending jobs still run) -- supersedes our QUEUE_CLOSED preason on the dispatch path; sched_queue_status_reason drops that branch - bug-1846: config push now stashes and applies between scheduling rounds (sched_stash_config/sched_apply_pending_config), applied at the same main-thread round boundary that preserves bug-1513 serialization - bug-1760: sched_check_pending_rejoin exposed + xcecli_is_connection_broken public predicate - bug-1936: execuser/exechome/execcwd/exdata carried on submission (superset of our bug-1500 execuser fallback; our job-typed resolveExecUser removed as dead code) - bug-1945~1962: apiserver auth hardening (HS256 pinning, default- secret rejection, logout revocation, bcrypt, RBAC) -- our API-key branch folded into controllers.ParseTokenClaims so both coexist - bug-1736: rawRunUsage job run-usage fields in the Go model - xce_test relocated src/controller/testsuite/test.c -> src/harness/test.c; controller suite split into controller_test.c + fixtures; their clitest fixtures spawn xce_test mid-suite under `meson test -j` Ours preserved across the merge: - PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) -- gate re-ported into their relocated allocator (sched_election.c) after plugin_job_election, with attach on the allocator thread only (bug-1521 race discipline) - PEND-reason P0 main_reason semantics RECONCILED to theirs: the merge initially re-carried our first-detail-add-promotes-main_reason fallback in add_job_preason (4e2ac86), but their whole trace/verify stack treats "detail-only => main_reason 0" as a contract -- sched_trace_capture counts main_reason as its own occurrence ("Expected 2 Was 3" in test_trace_capture), the outcome renderer prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e), and the committed .replay/.sim_verify baselines encode main_reason=0 (test_repl_trace_baseline byte compare, test_trace_equivalence off-vs-traced). The fallback is dropped from the shared lib; the user-visible "overall reason" survives as an API-layer view: rawJobToModel (convert_pure.go) promotes the first detail row to main_reason_name when main_reason is 0 (main_reason VALUE marshals unchanged). add_job_preason keeps only the NULL-reasons guard - bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs - bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant (their jobCtrlUnionDiscriminant, same semantics) - xcecli typed PendingReason field (53c00834), bug-1522 reconnect resend, xcecli_force_reconnect + their is_connection_broken both exported in client.h/libclient - t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494), /xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2) - lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block - drop-audit: our bug-1504 fix covered by their bug-1680 (no action) bug-924 follow-up (replaces the67f3f6adscratch-dir wipe): the take-theirs audit's one "drop" was re-evaluated and REVERTED instead of re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under `meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as a child mid-suite; the wipe destroyed the parent's fixtures -> test_test_cli failures). Coverage instead: - mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since (testsuite_support/mkdtemp_clean.[ch]) - new case_scratch_sweep.[ch]: aggregate runners (controller_test.c, harness_test.c -- hundreds of cases per process) call case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's dirs are swept immediately in tearDown (after engine reaping), FAILED cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1 disables. One full controller_unit_test run was observed to leak 12G+ to ENOSPC without this, cascading into unrelated "Expected Non-NULL" failures - harness_unit_test_deps links the shim library as well Test reconciliations on adopt: - core-ready-gates: sched_queue_is_ready now gates only INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect -- covered end-to-end by test_queue_admin_control.c - test_worker_resync: the 4 restart-replay RUN assertions predate the bug-1509 B3 marking (8858afed) landing on the same lineage; replayed RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN); terminal DONE/EXIT assertions unchanged - convert_pure.go: our job-typed resolveExecUser deleted (superseded by their bug-1780/1936 two-arg version) - test_preason: the 3 P0-promotion assertions reverted to the detail-keeps-main-0 contract; new tests cover limit-code detail rows and explicit-main-wins-over-later-details; apiserver convert_test.go gains the detail-only name-promotion cases Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen
commit
0c8607345e
663 changed files with 139848 additions and 13614 deletions
|
|
@ -62,11 +62,34 @@ xce_select_parser = custom_target(
|
|||
],
|
||||
build_by_default: true)
|
||||
|
||||
libxce_src = [ gen_srcs[0],
|
||||
section_parser[0],
|
||||
rusage_parser[0],
|
||||
xce_resreq_parser[0],
|
||||
xce_select_parser[0],
|
||||
libxce_src = [ # bug-1798: pass the whole gen_srcs custom_target (both
|
||||
# controller.capnp.c AND controller.capnp.h), not just
|
||||
# gen_srcs[0] -- several files below (utils.c and others)
|
||||
# reach controller.capnp.h only via #include, which ninja
|
||||
# cannot see. Indexing to gen_srcs[0] dropped that header
|
||||
# from this target's declared outputs entirely, so a
|
||||
# from-scratch build had no dependency edge forcing capnp
|
||||
# codegen to finish before those files compiled,
|
||||
# intermittently failing with "controller.capnp.h: No such
|
||||
# file or directory" (recurring since 2026-07-16). Passing
|
||||
# the full custom_target compiles only its .c output (as
|
||||
# before) while ordering the ENTIRE target's build after
|
||||
# BOTH outputs are generated.
|
||||
gen_srcs,
|
||||
# bug-1798 (same class): each packcc parser custom_target
|
||||
# below also produces a paired .h output (section.h,
|
||||
# rusage.h, xce_resreq.h, xce_select_peg.h) that only
|
||||
# resreq.c #includes -- a separate file from each parser's
|
||||
# own generated .c. Indexing to [0] dropped every one of
|
||||
# these headers from libxce_src, leaving resreq.c's compile
|
||||
# with no declared dependency on any of the four codegen
|
||||
# steps. Passing each custom_target unindexed compiles only
|
||||
# its .c output (as before) while ordering this whole
|
||||
# target's build after both of its outputs.
|
||||
section_parser,
|
||||
rusage_parser,
|
||||
xce_resreq_parser,
|
||||
xce_select_parser,
|
||||
'client.c',
|
||||
'client_failover.c',
|
||||
'dataq.c',
|
||||
|
|
@ -82,6 +105,7 @@ libxce_src = [ gen_srcs[0],
|
|||
'window.c',
|
||||
'xceapi.c',
|
||||
'xcectx.c',
|
||||
'xcelist.c',
|
||||
'xlist.c',
|
||||
'xstring.c',
|
||||
'zarray.c',
|
||||
|
|
@ -137,7 +161,17 @@ libxce = static_library('xce',
|
|||
install: true)
|
||||
|
||||
|
||||
libxce_dep = declare_dependency(link_with: libxce)
|
||||
|
||||
# bug-1798: `sources: [gen_srcs]` forces every consumer of libxce_dep
|
||||
# (controller/worker/scheduler, none of which otherwise declare any
|
||||
# dependency on this codegen step even though several of their own .c
|
||||
# files #include "controller.capnp.h" directly via a raw -Ilib path) to
|
||||
# build gen_srcs' outputs before compiling their own sources -- closing
|
||||
# the same missing-dependency-edge race as the libxce_src fix above, but
|
||||
# for consumers OUTSIDE the libxce target itself. link_with alone does not
|
||||
# propagate this: it only orders the LINK step after libxce is built, not
|
||||
# a consumer's own COMPILE steps against a header libxce never installs.
|
||||
libxce_dep = declare_dependency(link_with: libxce, sources: [gen_srcs])
|
||||
# libxce_shared_dep = declare_dependency(link_with: libxce_shared)
|
||||
|
||||
libxce_test_src = ['testsuite/test.c',
|
||||
|
|
@ -149,6 +183,7 @@ libxce_test_src = ['testsuite/test.c',
|
|||
'testsuite/test_zarray.c',
|
||||
'testsuite/test_config.c',
|
||||
'testsuite/test_idgen.c',
|
||||
'testsuite/test_jobsel.c',
|
||||
'testsuite/test_delay.c',
|
||||
'testsuite/test_ordq.c',
|
||||
'testsuite/test_preason.c',
|
||||
|
|
@ -159,6 +194,8 @@ libxce_test_src = ['testsuite/test.c',
|
|||
'testsuite/test_limit.c',
|
||||
'testsuite/test_xvec.c',
|
||||
'testsuite/test_resreq.c',
|
||||
'testsuite/test_resreq_fingerprint.c',
|
||||
'testsuite/test_resreq_task_count.c',
|
||||
'testsuite/test_resmgr_fixture.c',
|
||||
'testsuite/test_select.c',
|
||||
'testsuite/test_bitset.c',
|
||||
|
|
@ -185,12 +222,18 @@ libxce_test_src = ['testsuite/test.c',
|
|||
'testsuite/test_netio_close_propagation.c',
|
||||
'testsuite/test_netio_bounded_connect.c',
|
||||
'testsuite/test_netio_batch_dispatch_uaf.c',
|
||||
'testsuite/test_netio_err_queue.c',
|
||||
'testsuite/test_netio_nodelay.c',
|
||||
'testsuite/test_request_send_propagation.c',
|
||||
'testsuite/test_raft.c',
|
||||
'testsuite/test_wal_wire.c',
|
||||
'testsuite/test_ctrlstatus.c',
|
||||
'testsuite/test_vtask_admission.c',
|
||||
'testsuite/test_vtask_enq_race.c',
|
||||
'testsuite/test_vtask_pkt_handoff.c',
|
||||
'testsuite/test_ctx_reconnect_quiesce.c',
|
||||
'testsuite/test_vtask_order.c',
|
||||
'testsuite/test_vtaskmgr_teardown.c',
|
||||
'testsuite/test_xcealloc.c',
|
||||
'raft/raft_sim.c',
|
||||
]
|
||||
|
|
@ -210,3 +253,22 @@ libxce_test = executable('xcelib_test',
|
|||
|
||||
test('libxce test', libxce_test, timeout: 120,
|
||||
env: ['ASAN_OPTIONS=detect_leaks=0']) # suite convention — fixtures not torn down
|
||||
|
||||
# bug-1882 regression: `-Wl,--wrap=strdup` linker symbol interposition (same
|
||||
# idiom as bug-1843's sched_record_lifecycle_test `--wrap=realloc`) so the
|
||||
# tests can make a chosen strdup() inside make_alloc()/make_host_alloc()
|
||||
# fail. Its own binary, so xcelib_test's other tests never see the wrap.
|
||||
# libxce-alloc-preason-equality: it also wraps vasprintf (and the
|
||||
# _FORTIFY_SOURCE __vasprintf_chk variant) to fail the equality functions'
|
||||
# description allocation.
|
||||
libxce_alloc_oom_test = executable('xcealloc_oom_test',
|
||||
['testsuite/test_xcealloc_oom.c'],
|
||||
include_directories: [incdir, build_inc],
|
||||
dependencies: libxce_test_deps,
|
||||
c_args: ['-DXCE_TESTING'],
|
||||
link_args: ['-Wl,--wrap=strdup',
|
||||
'-Wl,--wrap=vasprintf',
|
||||
'-Wl,--wrap=__vasprintf_chk'],
|
||||
install: false)
|
||||
|
||||
test('libxce alloc OOM test', libxce_alloc_oom_test, timeout: 60)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue