Pure verification, zero code/test changes. Confirmed: the 9 pre-existing
test_sim_reconcile_unaccounted.c tests pass unchanged; interactive/batch
parity (test_repl_equivalence.c, untouched by this spec) passes; full
scheduler suite shows the identical 53 Ok / 1 Fail baseline (sole failure
the pre-existing, unrelated bug-1560), with the spec's 6 net-new tests
(tasks 1.1/1.2/2.1/3.1/3.2) as the only addition to that baseline.
Marks all tasks in the scheduler-sim-eager-reconciliation spec complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_sim_verify_no_false_mismatch_across_round_separated_dispatch:
a job's DISPATCH decision at round 1, RUN-state confirmation event at
round 4, --sim-verify enabled throughout, asserting zero mismatches.
Fixture correctly accounts for a mechanically-verified system behavior:
sim mode's live picker (sim_picker_thread_fn/builtin_fcfs) never emits
on_dispatch() -- it unconditionally re-PENDs every candidate each round,
including a job reconciliation already captured, until the job's
recorded JOB_STATE_CHANGE event removes it from pendJobs. The fixture
includes a matching PEND(jobid) entry at each intervening round to match
this reality, not an assumption.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap:
a worker at exact capacity (jobs A=8.0 + C=2.0 against 10.0 capacity), with
each job's RUN-state confirmation landing several rounds after its DISPATCH
decision (round 1 dispatch, round 4/5 confirmation, deliberately different
gaps). Drives the real sched_check_host_satisfy() admission check at three
checkpoints, proving Requirement 1 (1.1-1.4) at the outcome level -- a
capacity check, not just a resInUse display value -- per
scheduler-driver-equivalence.md's explicit contention-fixture discipline.
Round 1 review caught three new xce_sched_sim_driver_t fixtures declared
stack-local, amplifying an already-documented (bug-1598, bug-1632)
stack-use-after-return class from the never-joined picker thread
sched_sim_driver_run() spawns. Fixed by declaring them static (safe: each
is written by exactly one call site, and sched_sim_driver_start() memsets
the struct first) -- independently confirmed via 3 full-suite runs with
zero ASAN hits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds 3 tests proving guarantees that must hold under the widened
capture-eligibility gate: an already-captured job is never re-searched
or re-captured regardless of state (test_reconcile_still_skips_already_
captured_job_regardless_of_state), a PEND job with no match this round
stays uncaptured and remains eligible next round (test_reconcile_
no_match_leaves_pend_job_uncaptured_and_retries), and several
simultaneously outstanding jobs are captured independently with zero
cross-job interference (test_reconcile_multiple_jobs_captured_
independently_across_different_rounds).
Required making the shared add_job() fixture helper tolerant of -EEXIST
on a repeated queue registration (needed for the multi-job test, the
first in this file to call add_job() twice against the same core) --
traced to kvtab_add()'s narrowly-scoped -EEXIST return, provably a
no-op for all 11 pre-existing single-add_job()-call tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_captured_pend_job_excluded_from_resinuse_until_run: drives a
job captured while still PEND through sched_account_jobs() and confirms
its resource-in-use contribution is zero; transitions the same job to
RUN and confirms sched_account_jobs() now includes its exact captured
allocation, with no second reconciliation call in between -- proving the
nonzero value comes from the earlier capture, not a fresh one.
Required promoting the file's sched_account_jobs() from a no-op stub to
a real reimplementation (verbatim body, matching the exact precedent
already established in test_sim_contention_parity.c and
test_sim_round_boundary_reconciliation.c) plus a resource.c white-box
include -- no production code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Widens sim_reconcile_unaccounted_jobs()'s trigger gate in sim_loop.c: removes
!JOB_IS_RUN(job->state) so a job's recorded DISPATCH decision is captured as
soon as the replay reaches the round it occurred in, regardless of the job's
current state -- closing bug-1611/bug-1613's residual (a job whose RUN
confirmation lands in a later round than its dispatch decision, the realistic
case under any real dispatch-to-confirmation latency, was never reconciled).
sched_account_jobs()'s own unmodified JOB_IS_RUN charging gate continues to
defer actual resource-in-use accounting until the job genuinely transitions
to RUN -- no other line in the match/capture/report path changes.
Replaces test_reconcile_skips_non_run_state_job (which asserted the bug as
correct behavior) with test_reconcile_captures_non_run_state_job_when_dispatch_recorded,
proving the opposite for a PEND-state job. Adds the xce-test-rc-file-style
spec docs (requirements/design/research/tasks) for this feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
_live_e2e_common.py's CONTROLLER_CANDIDATES/XCEADMIN_CANDIDATES only ever
resolved to the three default profile-dir names (xce.build, build.release,
build.dbg), with no way to point the live E2E suite at a binary built to a
custom/explicit build dir. Adds XCE_CONTROLLER_BIN/XCEADMIN_BIN env-var
overrides via a new resolve_binary_candidates() helper -- when set, the
override becomes the sole candidate; otherwise the existing three-name
fallback list is unchanged. Mirrors the ${XCE_CONTROLLER_BIN:-default}
convention already used by clean_failover_preservation.sh/run-ha-soak.sh.
Skip messages in both live E2E test files now mention the override.
TDD: 4 new unit tests in test_live_e2e_common.py (no override, empty
override, override-becomes-sole-candidate, and an end-to-end reload proving
the real module constants respect their env vars). Full xceadmin suite
(288 tests) confirmed unaffected: 287 pass, 1 pre-existing unrelated
failure (a stale redirect-gap assertion, unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Now that xce_test auto-loads a personal rc file at startup (this spec),
harden the three existing non-interactive xce_test callers so a future
developer's ~/.config/xce_test/rc can never silently affect a CI/soak run:
run-ha-soak.sh, run-resource-verify.sh, and clean_failover_preservation.sh
all now pass --norc to their xtest() invocation. No behavior change today
(no rc file exists in this environment yet, per xce-test-rc-file task 3.3's
own verification) -- this is forward-looking reproducibility hardening.
Independently re-verified end-to-end: run-ha-soak.sh (--sequential, 2
rounds, zero job loss) and run-resource-verify.sh (1 resource, 3 layers,
zero mismatches) both PASS unchanged with --norc added.
clean_failover_preservation.sh could not be run in this sandbox -- its
DATADIR precondition (<repo>/ha/data/cluster.bin + certs) was deleted by an
unrelated earlier commit (ee8473bd, admin-config-tool) and never restored;
confirmed via git history this is pre-existing and unrelated to this edit
(the precondition check runs before xtest() and fails identically on the
unmodified script). Logged as bug-1721, not fixed here -- out of this
spec's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
No product/spec-test code changed. Ran the real xce-test-ha soak and
xce-test-resources verification drivers with no rc file present anywhere
in the environment they run under -- both pass their own documented
success criteria unchanged (zero job loss across rolling failover; zero
resource-value mismatches across controller/scheduler/apiserver), proving
Requirement 1.3's "no rc file -> unchanged behavior" guarantee holds for
real, pre-existing automation that has no awareness this feature exists.
Marks all tasks in the xce-test-rc-file spec complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Extends test_rcfile_integration.sh with 2 scenarios proving Requirements
3.1-3.4: a failing rc-file command in piped/scripted mode aborts before
any of the piped script's own commands run, exiting nonzero; the same
failing command in an interactive session still reports the failure,
continues executing the rc file's remaining lines, and reaches the normal
prompt -- while still carrying the session's sticky nonzero exit-code
verdict (g_session_failed is never cleared, matching how any other
failing-command-then-quit interactive session already behaves).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_rcfile_integration.sh, mirroring clean_failover_preservation.sh's
shell-driven black-box pattern against the real built xce_test binary:
- a valid rc file's set masters override takes effect in both a piped run
and a genuine pty-driven interactive run, before the normal session begins
- --norc leaves the default masters value untouched with the same rc file
present
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds the integration call site in main(), between g_continue_on_error's
computation and the read loop: unless --norc is given, resolve the rc-file
path and load it through preprocess_and_run (the same command-execution
path used for typed/piped input). A NULL path (no XDG/HOME) or missing
file falls through silently to the read loop. A failing rc-file command
under scripted/piped abort-on-first-failure mode short-circuits straight
to the existing g_session_failed exit-code convention, never entering the
read loop -- so a piped script's own commands never run after an rc-file
failure. Verified end-to-end against the built binary, including a real
pty for the interactive path, in both the failure and success cases.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds struct arguments.norc, a --norc argp option, OPT_NORC (matching
main.c's OPT_PEERS/OPT_CONF_KEEP_N idiom), and its parse_opt case to
test.c. Parsing/recording only -- no wiring into main()'s control flow
or rcfile.c yet (task 2.2).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds rcfile_load(path, runner) to rcfile.c/rcfile.h: reads lines with
getline() (no fixed-size buffer, so a long line like a multi-peer
`set masters ...` can never be silently truncated or split), strips the
trailing newline, invokes the caller-supplied runner once per line, and
stops at the first nonzero runner return, propagating that value. A missing
rc file is a silent no-op; any other open failure prints one stderr warning
and returns 0.
Found during GREEN: fopen() succeeds on a directory path on this platform,
so the EISDIR failure only surfaces on the first getline() call (ferror(f)
set) -- added a post-loop ferror() check to catch it (bug-1716).
5 new unit tests (11 total with task 1.1): ENOENT-silent, per-line-in-order
with newline stripped, stop-on-nonzero with exact propagation, non-ENOENT
open-failure warning, and a >4KB single line proving no truncation/split.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds rcfile_resolve_path() (rcfile.c/rcfile.h): resolves $XDG_CONFIG_HOME/xce_test/rc
when set and non-empty, else $HOME/.config/xce_test/rc, else NULL. Standalone
libc-only module mirroring the configcompare.c pattern; wired into xce_test_src
and controller_unit_test_src, registered in controller_test.c's Unity runner.
6 unit tests in test_rcfile.c cover all XDG/HOME combinations plus a dedicated
env-var save/restore isolation test, addressing the cross-test leak risk found
during design review (controller_unit_test runs 80+ test files in one process).
Adds the xce-test-rc-file spec (requirements/design/tasks/research), including
design fixes from validation: getline()-based line reading (task 1.2, avoids
truncating long rc-file lines) and the env-isolation requirement above.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Merge follow-up fixes found during comprehensive unit-test + live-cluster
verification of the scheduler-framework merge:
- apiserver/backend/clib.go (ControlJob): Rongsong's clib.go had reverted
our bug-1490 fix — ctrl.action must be the capnp JobControl union
DISCRIMINANT (0=siginfo,1=ckptoption), not the models.JOB_* constant
(1/2/3/8), or every kill/stop/resume fails -EINVAL. Restored
ctrl.action = C.int(jobCtrlDiscriminant(action)); verified live (job
kill RUN->EXIT).
- controller/persist.c (fsync_worker): the merged bug-1563 no-progress
backoff used the PRE-FLUSH `pending` snapshot, so a no-op flush (another
path advanced durable_lsn to tip mid-flush) also slept wal_sync_ms and
raced commit_engine_fini in teardown, hanging the p2_integration suite
~60% (bug-1515). Re-evaluate the level AFTER the flush (pending_after)
so only a genuine sustained fdatasync failure backoffs. 20/20 clean p2
runs (was 3/5 hang). Also carries the bug-1509 B3 RUN->UNKWN marking in
persist_recover.
- scheduler/worker testsuite meson.build: add unity_dep parent includedir
(unity.pc Cflags only add -I${includedir}/unity, so <unity/unity.h> was
unresolvable) — same fix as src/lib/meson.build.
- test_handlers.c: re-point notify.data.dispatchfail after the first
sched_handle_notify NULLs it on success (code=0 branch was -EINVAL).
- test_controller_connector.c: give the rejoin-success fixtures a valid
worker_list pointer (my bug-1509 resend_job_status deref was NULL-
crashing).
- test_wal_job_dispatch_fact.c / test_dispatch_commit.c: Rongsong's
WAL_JOB_DISPATCH replay tests asserted JOB_STAT_RUN after persist_recover,
conflicting with bug-1509 B3 (RUN->UNKWN on restart). Assert JOB_IS_RUN
(base RUN recovered, incl. bug-1678 allocs) + UNKWN flag instead.
All suites green: controller 1145/0, scheduler 57/57, worker 15/15,
lib 303/0, apiserver CGO build ok.
Found via the mandatory live HA chaotic soak for this session's controller
audit campaign -- unrelated to any of the 9 audit fixes (bug-1684/1687/
1689/1690/1691/1692/1693/1694), all in ha.c/persist.c/job.c/cluster.c/
sched.c/worker.c; this is in myapp.c, untouched by any of them.
myapp_run()'s bootstrap "is a leader already known" check had a fallback
`c.ha.raft.term > 0` meant to detect genuine new raft activity processed
during the startup-stagger nap. But ha_restore_term() loads ha->term from
disk BEFORE this check runs, on every boot -- and Raft terms only ever
increase, never reset. So `raft.term > 0` is trivially true on every
restart for the rest of the cluster's on-disk lifetime, once it has ever
held one election, regardless of whether any leader is currently
reachable. Live-reproduced: every node, on every restart after the
cluster's first election, unconditionally skipped ha_start_election_async
forever -- no node could ever re-assert leadership through a real
election again. bug-1477's existing cluster_init_phase2 retry loop kept
firing correctly (confirmed in the reproduction logs) but could never
succeed, since the deeper cause -- no election ever runs to produce a
live primary to push config from -- was never addressed by that fix.
Fixed by snapshotting `term_at_boot` right after ha_restore_term()/
ha_init_raft() seed the term, strictly before the stagger nap, and
comparing `raft.term > term_at_boot` instead of `raft.term > 0` -- only a
term that has genuinely ADVANCED during this boot cycle (real vote/
heartbeat traffic processed) counts as evidence a leader is known. The
decision was extracted into a new, unit-testable myapp_leader_already_known()
(startup_validation.c, since controller_unit_test deliberately excludes
myapp.c/main.c). voted_for_name/vft==cur and primary_ctrl_fd>=0 checks
are unchanged.
7 new tests in test_ha_startup.c; RED/GREEN independently verified.
Rigorous A/B live-soak proof (identical 6-round --chaotic run, fixed vs
git-stashed-back-to-original): baseline showed term frozen for the whole
run and zero election activity on any non-primary node; fixed run showed
term genuinely progressing and ctrl-2 achieving a real, log-verified
CTRL_PRIMARY transition via a won PreVote round -- proof a node other
than the original primary can now take over, which was structurally
impossible before. Full controller suite: 1191 tests, 2 failures -- both
the pre-existing, unrelated bug-1561 overlayfs sandbox flake.
Residual, NOT independently verified yet (flagged honestly, not fixed
here): the fixed run's chaotic verdict showed committed_job_loss=6
(baseline 0) -- traced via waldump to one jobid missing from every node's
WAL, at the cold-boot window before round 1. Resembles the already-fixed
bug-1487 false-positive class (apply-pump-lag racing verify_classify's
expect-job check) but the reported mechanism doesn't perfectly match
(bug-1487's job was present-in-WAL/not-yet-visible; this one is described
as absent from every WAL entirely) -- needs independent re-verification
before being dismissed, tracked as a follow-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the 9 confirmed findings from the /xce-audit-invariants controller
pass. checkpoint_task_fn was registered directly as a delaymgr entry and
ran its full blocking body inline: checkpoint marker pwrite+fdatasync,
write_snapshot()'s genuinely unbounded per-entity walk (scales with
cluster state size) held under apply_pump_lock, WAL trim, and snapshot
rotation/unlink -- all on the same thread that drives heartbeats/election
timeouts. Unlike every other delaymgr operation (snapshot send,
config-push, reconnect, catch-up), checkpoint had no offload mechanism
at all -- the one gap left from the raft-drive-async hardening pass.
Design: full offload via the SlowLane (slowlane_submit), default ON --
NOT the vtask pool. persist_apply_pump_run() already holds apply_pump_lock
across its own per-entry walk while running ON the vtask pool
(VTASK_LIVE_BLOCK); putting checkpoint's unbounded, lock-held
write_snapshot() on that same pool risked starving it, exactly what
ha-async-io.md's own thread-role table warns against. The SlowLane is
the actual established pattern for this I/O class (push_config_transport_only,
the snapshot-send path, catchup_lane_fn, demotion_lane_fn all dispatch
there) -- and persist_checkpoint() already runs on the slow lane in
production TODAY via ha_snapshot_do_send() -> snapshot_lane_fn, so
apply_pump_lock's acquisition from a slow-lane thread is proven-safe,
exercised daily, not new surface. Unlike catchup/demotion (which stay
flag-gated OFF for an unrelated, still-not-root-caused hang, bug-1429),
checkpoint's new flag defaults ON given this reuses an already-proven
code path. Inline-fallback + WARN only on a genuine slowlane_submit
failure (not -EAGAIN/-EEXIST, both non-failures) -- never silently drops
a scheduled checkpoint.
New test_persist_checkpoint_offload.c (3 tests). RED-phase forced the
pre-fix always-inline path: the suite never printed a single result and
had to be killed, proving the test genuinely detects the violation, not
a tautology. GREEN: all 3 pass. Full suite: 1184 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake
(exact signature match). No live HA soak was run for this specific fix
in isolation (explicitly scoped out, not hidden) -- the mandatory
live-soak proof for this whole audit campaign covers it next.
This closes the last of the 9 confirmed findings from the controller
audit (bug-1684, bug-1687/1689/1690/1691/1692, bug-1693, bug-1694).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."
OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.
Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.
schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.
Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the five field-parity gaps from the original audit (bug-1678,
bug-1687, bug-1689, bug-1690, bug-1691). set_preason()/clear_preason()
set job.pend_reasons (host/resource-based pending-job explanations), but
persist_job_hook_fun's default: arm only ever captured {jobid, state}.
Neither apply seam restored pend_reasons after a restart.
Reuses the ALREADY-EXISTING JobPendReason/HostBasedReason/ResourceBasedReason
capnp structs -- no schema change needed, unlike bug-1690's WAL_JOB_RUSAGE.
Safety-checked against bug-1690's lesson (RTTI/makeString()-backed fields
need merge-not-replace, not free-and-replace): pend_reasons is plain
malloc/strdup-backed throughout (erase_job_preason is FREEUP()-only,
add_host_preason/add_res_preason use strdup()/calloc()), so a
straightforward free-and-replace is safe here, matching bug-1678/1687's
pattern instead.
New wrinkle beyond the four prior fixes: pend_reasons has a genuine CLEAR
operation, and "trailing bytes present or absent" can't distinguish a
legitimate clear from an old-format payload -- added an explicit presence
byte (0=cleared/adopt NULL, 1=capnp bytes follow), verified by two
dedicated "clear survives restart" tests.
11 new tests in test_wal_job_preason_fact.c; RED/GREEN independently
verified (7 genuine pre-fix failures, one test's own wrong premise about
capnp's garbage-byte behavior caught and corrected during RED/GREEN
iteration). Full controller suite: 1178 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
This closes the field-complete apply parity invariant class the
xce-audit-invariants controller dimension 7 was added for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
After merging origin/scheduler-framework (scheduler-core), re-apply and
reconcile our fixes that Rongsong's branch lacked or implemented differently:
- handler.c: keep the code<0 dispatch-fail check (bug-1512: a code-0
success ack must NOT re-PEND, or dispatched jobs spawn duplicates).
Use sched_job_state_change (Rongsong's enqueue mechanism) instead of
sched_requeue_pend_job so the merged test framework (pop_jstate_change)
stays valid. Drop Rongsong's duplicate unconditional re-PEND case.
- scheduler.c: port sched_requeue_pend_job (bug-1494 class) adapted to
sched_get_job(&sched->core, ...) signature.
- types.h: re-add xce_sched_queue_t.dispatchwindow (our time-window PEND
reason field) dropped when taking Rongsong's types.h.
- models/jobs.go: re-add JobSubmitResponse.JobIDHuman (human job id).
- worker controller_connector.c: our bug-1509 resend_job_status on rejoin
success was already ported; add test stub.
- tests: adapt dispatch-fail test (code=0 → no re-PEND); fix sched_get_job
signature in white-box stubs.
Build passes (nix build .#xce). Unit + live-cluster verification pending.
Co-Authored-By: Claude <noreply@anthropic.com>
