One logger.Debug("resource-location stored", ...) call in
OnResourceLocationUpdated, immediately after store.ResourceLocations.Put,
logging resource/host/value as zap fields. Debug-level only, matches this
package's existing logging convention (reuses convert.go's field names).
Live-confirmed against a real deployment with logging.level: debug.
Also surfaced (but did not fix, out of boundary - pure Go change, no
possible causal link) a pre-existing xce_test segfault on string-typed
resupdate (type/model) - uninitialized stack memory, non-deterministic,
invisible to ASan. Logged as bug-1574; flagged in tasks.md as a risk for
task 7.1's default resource set and task 8.1's live e2e run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One TRACE (tlog_debug3) line in xce_core_push_events()'s RESVALS_DELTA case,
logging host/resource/value in a fixed format ("resval applied host=...
resource=... value=..."), immediately after the value is written to the
worker's live state. Debug-level only, no control-flow change.
Explicit svalue/fvalue branching (round-2 fix after review) - never reads a
Value_u_t union member without confirming .which matches first, since
Value_u_bvalue (boolean resources) is reachable at this path in production
and would otherwise be misread as a garbage float.
Live-confirmed against a real deployment at -vvv. Also surfaced (but did not
fix, out of boundary) a pre-existing double-free crash on a resource's
second periodic report - logged as bug-1572.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a
pre-built /tmp/apiserver binary, not built by this script) that generates
start-apiserver.sh with a debug-level logger.yaml, plus an always-generated
start-all.sh sequencing controller-health-check-then-worker/scheduler
(-vvv)/apiserver. Purely additive - callers without a built apiserver
(e.g. xce-test-ha) see zero behavior change.
Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the
step; with-apiserver reaches full backend registration with debug logging
active and scheduler emitting DEBUG3 output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cmd_resupdate() now issues a follow-up xcecli_resval() query per submitted
(resource, worker) pair after the update RPC, since update_resval_list()'s
reply can't carry per-item status. Reports accepted/rejected per name, with
a bounded retry (5x100ms) against propagation races.
Live-confirmed: `resupdate -n mem -v 5 -n not-a-real-resource -v 1` reports
mem accepted and not-a-real-resource rejected in the same invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
set_resval() now delegates type resolution to resval_typeresolve.c; unknown
resource names default to numeric, overridable via a new -t <numeric|string>
flag. cmd_resupdate()'s getopt loop defers per-pair finalization so a
trailing -t still applies to its pair, while preserving the exact original
duplicate-value rejection behavior (round-2 fix after review caught a
last-write-wins regression in that refactor).
Live-confirmed: resupdate -n r15m -v 1.5 (previously silently dropped) now
succeeds and is visible via resval.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move set_resval()'s resource-name lookup table into a new, dependency-light
resval_typeresolve.c/.h module so it's independently unit-testable (test.c
can't be white-box included into Unity). Adds RED tests for the not-yet-
implemented numeric-fallback/type-override behavior (task 2.1).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
No job could ever be scheduled in normal mode -- every round produced
decision=(nil), forever. Three independent bugs were stacked on the
same symptom, each masking the next (only reachable once the prior
one was fixed):
1. scheduler.c's sched_init_queues() -- the scheduler's one-time
synchronous initial-config-push handler -- only fired the legacy
job_elect_fn-era plugin_queue_update() fan-out. builtin_fcfs (the
default picker) has no .hooks, only picker_iface, so
on_queue_update() was never called for any queue from the initial
cluster config. Its own candidate registry stayed empty forever,
so next_job() found zero candidates on every session.
2. Once #1 was fixed and a job first reached the allocator,
core_events.c's core_build_job() never constructed job->schedobj
-- a documented but incorrect assumption from the scheduler-core
rewrite ("no picker hook reads job->schedobj... belongs to a
separate allocation spec" -- no such spec exists in the live
code). sched_allocator() dereferences it directly with no NULL
check; reproduced live as a SIGSEGV in limit_job_election().
3. Once #2 was fixed and a job reached genuine dispatch,
resource.c's put_resvals() crashed during post-dispatch resource
charging: `idxs[i] <= 0` treated resource index 0 ("slots",
almost always the first registered resource) as an invalid
sentinel, leaving its value slot NULL for an unconditional write
two lines later.
All three root-caused via gdb `bt full` on live SIGSEGVs, not
guesswork, and live-verified end-to-end: a 3-element job array now
runs PEND -> DISPATCH -> RUN -> DONE with the scheduler staying alive
throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
collect_load_indices() (resource_sync.c) tagged every periodic
mem/r1m/r15m/r15s/ut/pg/ls/it/swp/io/tmp report with gethostname()'s
raw OS/container hostname instead of the worker's own join identity
(xcecd_ctx.virtual_name, the -n/--name value). Every other worker-keyed
lookup downstream (scheduler's sched_get_worker(), and the same
HOOK_DETAIL_RESOURCE_UPDATE payload forwarded on to the Go apiserver)
matches by that join identity, so the mismatch silently dropped every
periodic resource report for the real worker.
Also adds a startup INFO log (main.c) naming the resolved job picker
plugin, used to rule out a legacy-picker hypothesis while chasing a
related (still-open) dispatch issue.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xce_scheduler -i -s <replay>: "info worker <name>" showed
total=0.00 available=0.00 for every resource, even though the live
cluster (xce_test resval/workerlist) reported real non-zero values
for the same worker.
Two independent root causes, both live-traced end-to-end against a
real deployed cluster (value-level TRACE at every hop: controller
push, scheduler apply, snapshot save, snapshot load/decode):
1. src/scheduler/src/core_events.c: XCE_CORE_EVENT_WORKERS_PUSH
called sched_add_worker() -- a plain key_add(), add-only --
unconditionally on every push. A worker connecting triggers
multiple WORKERS_PUSH events in quick succession (an early push
before the controller finishes writing real capacity values,
carrying stale/zero data, followed moments later by a push
carrying the real values). The FIRST push always won the race;
every push after it for the same worker name silently failed
-EEXIST and its resource values were discarded -- not just at
join, for the rest of the process's lifetime (unlike jobs, there
was no "update an existing worker" event path at all). Fixed via
a new core_upsert_worker() (mirrors the existing
core_upsert_queue() add-or-update precedent): looks up the
worker by name first, updates status+resvals in place if found,
else builds+adds a new one as before.
2. src/scheduler/src/repl_info.c: repl_info_worker_one()/
repl_info_resource_one() checked `xce_val_t.which ==
XCE_VAL_NUMERIC` -- but XCE_VAL_NUMERIC (xce/types.h, value 1)
is a resource-DEFINITION value-type constant
(xce_resource_t.valtype's own enum), a completely different enum
from xce_val_t.which's real domain, Value_u_which
(controller.capnp.h, Value_u_fvalue=5). Every producer in this
codebase sets xce_val_t.which to Value_u_fvalue, never
XCE_VAL_NUMERIC, so the comparison was always false -- every
genuinely-numeric resval displayed as 0.00 unconditionally,
independent of fix (1). Corrected all 4 occurrences to
Value_u_fvalue.
Both fixed via TDD (RED test before each fix, GREEN after):
test_core_events.c's test_workers_push_twice_updates_existing_
worker_in_place (two pushes, same name, different values -- both
must apply) and test_repl_breakpoints_inspection.c's
test_info_worker_reports_resource_values (a real numeric resval
must display, not fall back to 0.00).
Live-validated by reproducing the user's exact original scenario
end-to-end (deploy a real cluster, record, `xce_scheduler -i -s
<replay>`, `info worker <name>`) -- now shows slots=22.00,
maxmem=33182089216.00 (30.9GB), mem=17151664128.00 (16GB), matching
the user's own live xce_test resval/workerlist numbers exactly.
Full scheduler suite green (41/41, excluding pre-existing unrelated
bug-1560); controller suite green (1102/1104, the same pre-existing
bug-1561 pair, confirmed unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xce_scheduler -i -s <replay>: WARN "failed to resolve job picker
plugin (rc=-2)" on load, and every job already present in the
replay file's INITIAL snapshot stayed PEND forever -- never even
reached the picker, let alone got dispatched.
Root cause: sched_sim_driver_start() (sim.c) loads the initial
snapshot via xce_sched_core_load_snapshot() BEFORE any picker
plugin is registered (sim.h's own documented "Known scope
boundary" -- only the picker's NAME is threaded through from the
ReplayHeader). sched_resolve_job_picker() therefore always returns
-ENOENT there, and xce_sched_core_load_snapshot()'s own
picker-rebuild walk (on_queue_update/on_job_push per queue/job) is
silently skipped. The real builtin_fcfs picker only gets registered
moments later in sched_sim_driver_run() -- but nothing then replayed
that rebuild walk, so any job/queue from the INITIAL snapshot (as
opposed to one arriving later via a ReplayEvent, which correctly
reaches the picker through xce_core_push_events()'s own picker!=NULL
check) was permanently invisible to next_job().
Fixed by a new file, sim_picker.c: sim_bootstrap_picker() (replacing
sim_loop.c's sim_register_default_picker(), moved out because
sim_loop.c was already at its own documented 800-line "zero
line-budget headroom" cap) registers the picker AND then replays the
missed rebuild walk against it. Wired into scheduler_srcs,
sched_sim_isolation_lib, and all 9 white-box test TUs that #include
sim_loop.c directly.
New regression test (test_sim_event_loop.c): a job baked into a
fixture's INITIAL snapshot (not a ReplayEvent) now gets a live PEND
decision. RED confirmed by temporarily no-op'ing the rebuild walk
(test failed as expected), GREEN with the real fix. Full scheduler
suite green (41/41, excluding the pre-existing unrelated bug-1560).
Note: sim mode still never DISPATCHes (only PENDs) -- that is a
separate, intentionally out-of-scope gap (real resource-matching
allocation belongs to the not-yet-designed scheduler-allocation
spec), not addressed here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three related enhancements to make /xce-deploy-ha's output a
self-contained, directly-usable cluster for manual testing:
- xceadmin now builds an installable wheel (meson-build.sh) with
correct dependencies (click/pycapnp/PyYAML/jsonschema, verified
against actual imports -- pyproject.toml was missing several).
- deploy-ha.sh resolves the deploy environment's own bash
($BASH_BIN, baked into every generated script's shebang -- this
environment's deploy-ha.sh had drifted to #!/bin/sh while still
using bash-only syntax) and generates scripts/start-worker.sh /
scripts/start-scheduler.sh sharing the controllers' --peers, plus
a standalone xceadmin install (wheel extracted directly, resolved
$PYTHONPATH baked into a wrapper script -- nix devShell packages
are $PYTHONPATH entries, not merged into python3's own
site-packages, so a --system-site-packages venv can't see them).
- bug-1566: xceadmin's --controller option had no relationship to a
deployment's actual peers (hardcoded localhost:9991 default) --
every command required an explicit --controller or silently tried
the wrong port. Fixed via a $XCEADMIN_CONTROLLER envvar fallback
(click precedence: explicit flag > envvar > bare default),
exported by deploy-ha.sh's generated scripts/xceadmin wrapper.
Fixed/validated via TDD + live reproduction against a real deployed
cluster in a fully stripped environment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
setup_build() unconditionally rm -rf'd the whole profile build
directory on every single-component invocation -- running
/xce-build xceadmin after /xce-build controller silently deleted
xce_controller/xce_test/etc. from the same build dir.
Added an opt-in --incremental flag (meson setup --reconfigure in
place instead of wiping) rather than changing the default: this
project's capnp-generation custom_target has known missing-
dependency-edge races that a full wipe currently papers over as a
loud, retry-fixable failure. Making incremental the default risked
turning that same gap into silent stale-generated-code instead, so
the safer full-wipe behavior is preserved unless explicitly opted
into.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
persist_flush()'s synchronous fdatasync path advanced durable_lsn on
success but never called on_durable_advance() -- only the async
fsync_worker did. This gap was previously masked by bug-1563's own
busy-spin (a fast accidental repoll covered for the missing
notification); fixing that spin unmasked it, causing
test_durable_majority_commit_end_to_end to fail deterministically
(3/3 runs).
Fixed by firing on_durable_advance() from persist_flush() too,
whenever it genuinely advances durable_lsn. Fixed via TDD (RED test
before the fix, GREEN after) and re-validated with a fresh 40-round
sequential + chaotic HA soak (both pass cleanly, zero fdatasync
errors, zero divergence/loss).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.
bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.
Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xcecli_jobsubmit() could return rc>=0 (success) while leaving its *jobid
output parameter at 0, reproducible under concurrent client load
immediately after a fresh single-node election (~1-3/9 clients per round
in the original repro).
Root cause: XCECLI_FAILOVER_CALL's idempotent-replay rule
(client_failover.c's xce_failover_classify) normalizes a retried resend
that hits the server's EEXIST/inflight-collision rejection into overall
success -- but that rejection reply carries no payload
(handle_jobsubmit's EEXIST branches reply with RequestReply_data_nodata,
job.c), so xce_jobsubmit() never populates *jobid on the winning
attempt, nor did it on the earlier failing attempt (raw status was
already negative).
Adds jobsubmit_finalize_jobid() (client_internal.h/client.c), called
from xcecli_jobsubmit() right after XCECLI_FAILOVER_CALL returns. Sets
*jobid = submit->jobid (the client-generated id already on the wire --
the controller commits/echoes this exact value on every genuine success
reply, bug-786/job.c:1439) whenever the overall call succeeded,
otherwise leaves it untouched. Placed at the jobsubmit-specific wrapper
layer, not the generic classify/decide layer, which stays fully generic.
3 new socket-free unit tests in test_libxce_retry.c (following this
codebase's direct-state-simulation style for I/O-adjacent logic).
Live-validated: full ASAN rebuild clean, libxce test suite 1/1 OK with
all 3 new tests passing, and a 5-round/45-client concurrent-submit
repro against a live single-node cluster showed 0 "no clean ack"
failures (vs. 1-3/9 per round pre-fix).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds `xceadmin config dump CLUSTER_BIN_PATH`, the reverse of `config
build` -- decodes a serialized ClusterConfig back to human-readable text
so a built config can be inspected without a throwaway pycapnp script.
Motivated directly by how bug-1556 (hosts.yaml's max_slot: -1 auto-detect
sentinel silently becoming mxj=0) was actually found: by hand-decoding a
live cluster.bin. The dump renders the same sentinel as
"max_slot=-1 (auto-detect from CPU count)", not the raw uint32.
Also adds parse_size_or_number() to xcecommon.mapping so a queue/host's
sched_stop threshold on a byte-valued resource (maxmem/mem/tmp/maxswap/
maxtmp) can be written as a human-friendly size string ("500MB", "2GB")
instead of a raw byte count; a plain number still passes through
unchanged for non-byte resources like load averages. hosts.schema.json
and queues.schema.json's sched_stop schema gains a matching SizeOrNumber
type. dump_cluster_config() humanizes the same byte-valued thresholds
back on the way out (_format_bytes_human).
New ClusterBinFileError covers every local cluster.bin read/decode
failure mode without letting a raw OSError/capnp.KjException escape the
CLI. RESOURCE_VALTYPE/RESOURCE_ORDTYPE/RESOURCE_FLAG_* made public in
mapping.py so dump_cluster_config can reuse the same int<->name tables
build_cluster_config already uses, keeping both directions in sync by
construction.
256 xceadmin tests pass (incl. new test_config_dump.py and the
parse_size_or_number/SizeOrNumber cases in test_mapping.py/test_cli.py),
verified via `xce-build.sh xceadmin`'s full pinned out-of-tree install
+ pytest run before this commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.
Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.
Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apiserver now returns pending_reason as an object ({main_reason,
main_reason_name, host_reasons, res_reasons}) since the PEND-reason work.
xcecli's QueryJob.PendingReason was a plain string, so every job list
failed with "cannot unmarshal object into string". Two fixes:
- xcecli/models/jobs.go: PendingReason is now a string-backed type with a
custom UnmarshalJSON that accepts null / empty string / plain string
(older API) / the object form (current API, rendered as readable text —
LSF bjobs -p style: "match_fail; host t1: match_fail; ..."). Unknown
objects fall back to the raw bytes instead of erroring, so a job list
can never fail on an unexpected pending_reason shape.
- apiserver/models/jobs.go: JobAdditionalFields gains PendingReason so the
fields= query (xcecli `jobs list -l`) no longer rejects pending_reason
with "unsupported query field".
- xcecli/common/output/page_print_jobs.go: adapt the verbose printer to the
new type.
Tests: xcecli/models/jobs_test.go — 6 cases (null / empty / plain string /
object / within QueryJob / unknown object). xcecli 9 packages + apiserver
models/controllers/utils all pass.
Verified on the test cluster: `xcecli jobs list` (no unmarshal error) and
`xcecli jobs list -l` show "PENDING REASONS: host_unavail; host t1:
host_unavail; host t2: match_fail; host t3: match_fail".
Co-Authored-By: Claude <noreply@anthropic.com>
Covers the case where the worker process stays alive and only the TCP link
to the controller drops (or the controller fails over): libxce's failover
loop reconnects transparently and xcecli_hello() returns 0, so
checking_connection() sees ret==0 and never takes its
xcecd_join_cluster()/resend_job_status() path — job state changes that
occurred while disconnected are never re-reported, leaving stale RUN/UNKWN
jobs on the controller.
- worker/controller_connector.c: new xcecd_reconnect_cb registered via
xcecli_set_reconnect_handler after a successful join, resending the actual
job state (bug-1509 semantics) on every transparent reconnect.
- lib/client.c (bug-1522): xcecli_start no longer resets rejoin_cb/rejoin_arg
to NULL — it is initialized by xcecli_ctx_init's calloc, and a pre-start
registration (scheduler sched_reconnect_cb, worker xcecd_reconnect_cb) must
survive first connect; the failover loop invokes it after a transparent
reconnect (xce_ctx_reconnect), which never re-runs xcecli_start. Previously
every pre-start registration was dead code.
- scheduler.c: move xcecli_set_reconnect_handler to AFTER a successful join —
xcecli_join also runs the failover loop, and a candidate-iteration rotate
(standby redirect) must not fire the rejoin hook before the session exists
(was firing sched_reconnect_cb during initial join → rejoin on
uninitialized state → invalid-opcode crash).
- test_libxce_retry.c: regression test for bug-1522
(test_reconnect_handler_survives_start_state_reset). Suite 296/296.
Functional verification on the test cluster: worker joins (host OK), runs a
job, and after a controller-link break re-joins ("Rejoined cluster
successfully after retry N"). Scheduler stays up (no invalid-opcode crash).
Co-Authored-By: Claude <noreply@anthropic.com>
Add queue dispatch-window support so a queue whose window is closed
(queues.yaml dispatch_window, e.g. 19:00-08:30) is treated as
non-electable and reports PREASON_TIME_WINDOW.
- lib/window.c: new xce_timewindow_is_open() — NULL/empty always open,
same-day and overnight (close < open) segments, day-scoped (0=every day,
1-7=Mon..Sun).
- scheduler: make_sched_queue duplicates dispatchwindow_config onto
xce_sched_queue_t.dispatchwindow (freed by kill_sched_queue);
sched_queue_is_ready() checks it; sched_queue_status_reason() now takes
the queue (was a bare status bitmask) and maps a closed window to
PREASON_TIME_WINDOW.
- fcfs: keep the windowed-out queue skip; do NOT attach the queue-status
reason to job->schedobj from the picker thread (bug-1521: races the
allocator thread's MOVE_PTR(obj->reason,...) and corrupts the heap).
- tests: test_window.c (5 cases) covers no-window/daily/overnight/
day-scoped/multi-seg; full suite 295/295 pass.
Cluster-verified: with a 09:00-17:00 window on the xce queue (currently
outside the window), fcfs reports "queue xce ... ready=0 reason=8"
(PREASON_TIME_WINDOW).
Co-Authored-By: Claude <noreply@anthropic.com>
Two bugs surfaced by the new scheduler_pend_reasons tool:
1. The apiserver returns job_status as the numeric XCE state bitmask
(PEND=0x01, RUN=0x02, EXIT=0x04, DONE=0x08), but getJobs read it with
rowStr (string), which returns "" for a number → every job mapped to URM
"unknown", so the pending/running status filter matched nothing.
→ JobStatusFromXCEState(int) + rowURMStatus fallback.
2. pending_reason is a JSON object in the row, not a string; rowStr returned
"" so Extensions["pending_reason"] was empty and the reason never parsed.
→ rowJSON(key) stores the raw JSON value; the MCP pend_reasons parser
handles the map form.
schedulerclient tests updated (JobStatusFromXCEState table). mcp-server +
schedulerclient test suites pass; live MCP call now returns the bjobs -p style
explanation for a PEND job.
Co-Authored-By: Claude <noreply@anthropic.com>
New read-only tool that answers "why is my job waiting": lists PEND jobs (or
one by id) and translates each job's pending_reason into a human-readable
scenario (LSF bjobs -p style). Maps the PREASON_* codes surfaced by the
PEND-reason work (4e2ac86): no_resource / match_fail / limit_global /
limit_host / queue_closed / queue_draining / time_window / host_unavail.
- internal/pendreason.go: pending_reason JSON parsing (mirrors apiserver
rawPendReason) + reasonScenario() translation + bjobs -p style summary.
- internal/app.go: HandlePendReasons — id param for one job, else pending
list filtered by queue/owner/limit.
- cmd/main.go: register scheduler_pend_reasons (read-only).
- 5 unit tests (known/unknown codes, parse from JSON, missing reason,
summary format). go test ./... + go vet clean.
Co-Authored-By: Claude <noreply@anthropic.com>
P0 (done, cluster-verified):
- main_reason was always 0: add_job_preason only set it when BOTH host and res
were NULL, but the allocator always attaches host/res detail rows. Now the
first detail add wins as main_reason, so the API's main_reason_name is
meaningful (was "unknown(0)"). Verified live: PEND jobs now report
main_reason=3/match_fail + per-host reason_name.
- limit plugin attach: host-based quota hit (hg1_slots_per_host) -> LIMIT_HOST
+ limit name; global quota hit (ug1_slots, ...) -> LIMIT_GLOBAL + limit name.
Previously both blocks were silent (job just stayed PEND).
- Extended the PREASON_* enum (was 3 codes) with LSF-style classes:
LIMIT_GLOBAL/LIMIT_HOST/QUEUE_CLOSED/QUEUE_DRAINING/TIME_WINDOW/HOST_UNAVAIL.
apiserver pendReasonName maps them; host/res detail rows now carry reason_name.
P1 (implemented):
- queue status: sched_queue_is_ready no longer a stub — closed/inactivated
queues are not electable; the fcfs election attaches QUEUE_CLOSED /
QUEUE_DRAINING to every PEND job in a skipped queue.
- worker filter: sched_try_alloc_on_host_ skips UNAVAIL/CLOSED workers and
attaches HOST_UNAVAIL instead of trying to allocate on them.
Unit tests: preason suite updated for the main_reason change + 3 new tests
(limit host/global, first-add-wins); 295 lib tests pass under ASAN.
apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
The apiserver stayed circuit-breaker-degraded after a full controller
restart until manually restarted. Two root causes:
1. Breaker recovery was decoupled from the breaker itself. The breaker can
be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while
the controller is electing) with no heartbeat ever failing, but
StartHeartbeat only recovered on a fail→success heartbeat transition
(prevFailed), so the breaker stayed degraded forever.
→ extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when
prevFailed OR the breaker is currently tripped. 5 new unit tests.
2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's
internal ctx state is XCE_CTX_ERR. After a controller restart whose
election outlasts the FAILOVER_CALL retry budget, the retry loop's
xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the
FF path never reconnects and the client stays disconnected forever.
→ lib: new xcecli_force_reconnect() forces a rotate+recover regardless
of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while
degraded.
Verified on cluster: after controller kill+restart, the apiserver logs
"backend re-joined controller after reconnect" and job submissions resume
without a manual restart; a fresh job runs to DONE. Controller, scheduler
(rejoin ok, bug-1513) and workers all auto-recover.
Co-Authored-By: Claude <noreply@anthropic.com>
The "libxce reconnect heap corruption" (ASAN bad-free, free(): invalid
pointer) was NOT a libxce bug — xce_ctx_reconnect is fine. The crashes are
in the scheduler's rejoin state re-sync path (config/dataset re-push),
fixed with the reconnect/rejoin feature now integrated:
1. group.c makeGroup refresh: FREEUP(m->u.name) raw-freed shared interned
xstrings (makeString is a global refcounted table); the correct release
is dereference(), matching group_killer. → dereference().
2. limit.c limit_update_config: after key_add returns -EEXIST (rejoin
re-push — limit name already present) it called kill_limit(limit), but
xkvmgr_add's vclear already freed the limit via the kvmgr datakiller
(=kill_limit). → skip on -EEXIST (idempotent re-push).
3. handler.c/scheduler.c: the config push was applied inline on the vcpu
thread (sched_init_config adds queues/plugin queues) while job updates
flow through pend_updates on the main loop — concurrent rejoin re-push
let fcfs_update_job see a NULL j->queue (SIGSEGV). Route the config
push through pend_updates (XCE_SCHED_OP_UPDATE_CONFIG, killer case
already existed) and process config requests FIRST in sched_update via
a staging queue so jobs always see their queues/resources.
Verified: 295 lib unit tests pass under ASAN (incl. 2 new group-refresh
regression tests), scheduler survives 3 consecutive controller restarts
(each rejoin ok), full job lifecycle (submit→dispatch→DONE) on the
cluster. Also fixed stale build deps (libxce_deps missing librtti_dep,
scheduler_deps missing ssl_dep/c_capnproto_dep, unity include path).
Co-Authored-By: Claude <noreply@anthropic.com>
component-failover-adoption Req 4: when the apiserver's controller connection
is down, backend-requiring API requests must return a clear service-unavailable
error rather than hanging. Adds:
- health.go: an atomic circuit breaker (IsDegraded/MarkDegraded/MarkHealthy).
- callBackend: the four write calls (jobsubmit/jobctrl/queuectrl/hostctrl)
now route through it — degraded ⇒ immediate "backend controller unavailable"
error without invoking the lib; a transport OR XCE not-ready/masterless/
standby failure (E_XCE_NOMASTER/REDIRECT/NOT_READY/STANDBY) trips the breaker
so the NEXT call fails fast. Semantic rejections (-EINVAL/-ENOENT/E_XCE_BAD_*)
never trip it.
- StartHeartbeat: a failed tick marks degraded; recovery re-joins (rejoinBackend)
and marks healthy.
Verified live (t1 cluster): kill controller → first submit returns
xcecli_jobsubmit failed: -10013 (bounded, no hang), second submit returns
"backend controller unavailable (reconnecting)" in ~11ms. 7 unit tests.
Recovery-via-heartbeat after a FULL controller death is NOT yet reliable — the
lib transport reconnect does not always re-establish to the restarted
controller (same class as the scheduler reconnect; tracked under task 13).
Co-Authored-By: Claude <noreply@anthropic.com>
Closes the G-5 test gap mirroring the G-8 routing test: the rusage notify
handler enqueues XCE_SCHED_OP_JOB_RUSAGE and sched_update() must dispatch it to
sched_apply_job_rusage(), storing the report on the matching job (bug-765 G-5
remediation, production fix already in 93124c5). Driving the real sched_update()
dequeue loop with a recording stub, asserts the request reaches
sched_apply_job_rusage and the job's rusages field is set.
11/11 sched_redispatch_test PASS; full scheduler suite 6/6 OK. Live: job ran to
DONE on the deployed scheduler (G-3 leak fix in place), zero ASAN errors.
Co-Authored-By: Claude <noreply@anthropic.com>
component-failover-adoption Req 2 (partial): two of the three daemons now
self-heal after a controller restart without an operator restart.
- worker (controller_connector.c): checking_connection() previously gave up
after 5 retries and called handle_disconnect() (which set running=false and
exited the connector thread). It now retries indefinitely — the connector
thread is dedicated to this loop so the unbounded retry/sleep cannot block
the worker's other liveness duties. Verified live: killing the controller
leaves all 3 workers alive and retrying; they re-join when it returns.
- apiserver (clib.go): xcecli_heartbeat is fire-and-forget — it best-effort
reconnects the transport but never re-joins, so after a controller restart
push notifications stop. StartHeartbeat now tracks heartbeat failure→success
and, on recovery, calls xce_backend_rejoin (xcecli_join as HostRole_apiserver)
to re-register and resume pushes. Verified live: "backend re-joined
controller after reconnect" logged during a controller kill/restart.
Scheduler reconnect NOT included: the rejoin works but exposes a pre-existing
libxce race — xce_ctx_reconnect's endpoint_fini tears down the old endpoint
while a lib vtask/io thread is still processing it, a deterministic double-free
(bad-free in the loader, same freed address offset across runs). Root-cause in
libxce netio/endpoint lifecycle is a separate task (documented in
left_scheduler_work.md).
Co-Authored-By: Claude <noreply@anthropic.com>
Closes the msg-gap G-8 test gap the research flagged ("handler→dequeue ROUTING
is verified by code review, NOT an automated integration test"). The
remediation (UPDATE_JOB_FIELDS op routed to sched_update_existing_job) is
already in the code (93124c5); these tests lock the routing:
- test_update_fields_routes_to_inplace_path: driving the real sched_update()
dequeue loop, an UPDATE_JOB_FIELDS request reaches sched_update_existing_job
and mutates the existing job in place (same object, runtime state kept).
- test_bulk_update_existing_job_no_damage: a re-pushed dataset row for an
existing jobid is EEXIST-dropped by key_add (via the real sched_add_job) —
the live job keeps identity/state/runtime state.
10/10 sched_redispatch_test PASS; full scheduler suite 6/6 OK. Live cluster:
job PEND→RUN→DONE via notify updates, no WARN, no duplicate instance.
Co-Authored-By: Claude <noreply@anthropic.com>
The pending_reason field was declared on models.JobInfo but never populated:
scheduler computed PEND reasons and the controller stored them on
job->pend_reasons, yet the data stopped there. Two gaps closed:
1. controller/apiserver.c: apiserver_job_hook_fun now handles
HOOK_DETAIL_SETPREASON by pushing the updated job to API servers (reusing
apiserver_job_update_fun). Without this, a job whose PEND reasons change
never reached the API server — only JOB_NEW/UPDATE/START/EXIT events did,
and setpreason fired none of those.
2. backend/convert.go + convert_pure.go: cJobToModel / rawJobToModel now
extract xce_job_t.pend_reasons (main_reason + host/res reasons) and
marshal them into models.JobInfo.PendingReason as JSONB. models/jobs.go:
pending_reason added to JOB_DEFAULT_COLUMN so default job queries return
it (it was only in JSONBJFields, never selected).
Verified live on t1/t2/t3: a job with resreq "type == nosuchhost" stays
PEND and returns
"pending_reason":{"main_reason":0,"host_reasons":[{"host":"t1","reason":3},
{"host":"t2","reason":3},{"host":"t3","reason":3}]}
(main_reason=3 PREASON_MATCH_FAIL per host) via both /jobs list, /jobs/:id
and the SSE stream. 4 new unit tests (rawJobToModel pendReason marshal +
name mapping), all apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
Simplification per review feedback on e3b3ffa: a generic schema-driven
unit field touched the capnp wire schema (2 structs), C structs
(xce_resource_t/xce_resval_t), resobj.c's deep-copy/destroy, opget.c's
reply-build lookup, resources.yaml/resources.schema.json, and mapping.py
-- real cross-cutting cost for a mechanism only one resource (maxmem)
actually needs today.
Reverted all of that. The controller CLI (test.c) now just knows
"maxmem" is bytes-valued via a small explicit is_bytes_valued_resource()
name list, and adaptively picks the best-fit human unit at display time
(format_bytes_human, unchanged) -- same observable behavior (resval/
workerlist render maxmem as e.g. "30.90 GB"), zero wire schema surface.
Regenerated the xceadmin golden fixture back to its pre-unit-field size
(5304 bytes, exactly matching the size right after the mxj-only fix,
confirming a clean, complete removal).
Verified: 1095/1095 controller_unit_test (3 runs; one hit the
pre-existing flaky segfault already logged as bug-1553, unrelated),
228/228 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three bugs found while diagnosing a live controller log, all in the
worker resource-value path:
1. Every host relying on the documented hosts.yaml auto-detect default
(no max_slot, meaning "use CPU count") got 0 scheduling slots,
permanently blocking the scheduler from placing jobs on it. mxj is
UInt32 on the wire and can't carry the schema's -1 sentinel; mapping.py
worked around this by leaving it unset, which defaults to 0 -- and 0 is
never < 0, so apply_join_state()'s auto-detect fallback never fired.
Fixed by encoding the sentinel as 0xFFFFFFFF, which the controller's
`int mxj` field reinterprets back to -1 via its existing implicit
uint32_t->int conversion. No C-side or wire schema change needed for
this part; the fallback itself was already correct but had zero test
coverage for a negative mxj, now added.
2. update_resvals()/update_resval_list() logged the failure reason via
PERROR, which reads process-global errno -- but update_resval()'s
failure paths return raw negative codes directly and never touch
errno, so the controller log filled with misleading "update_resval:
Success" / "update_resval: Resource temporarily unavailable" lines
instead of the real reason. Fixed to decode the actual rc via
xce_strerror_r and name the failing resource.
3. maxmem is canonically stored/wired as raw bytes but had no unit
metadata, so resval/workerlist printed it as a bare number that read
as garbage (e.g. workerlist's fixed-width columns visually smeared a
multi-GB byte count across the MaxCPU/MaxMem headers). Added an
optional `unit` field to the Resource/ResourceVal capnp structs (and
xce_resource_t/xce_resval_t/resources.yaml) so resval and workerlist
can render a human-readable value (e.g. "30.90 GB") while the stored
value stays raw bytes -- display-only, every other resource unaffected.
Regenerated the xceadmin golden fixture (fixtures/full/expected.bin)
twice: once for the mxj sentinel encoding change, once for the new
Resource.unit field growing every resource's wire size by one pointer
slot. Both diffs verified byte-for-byte isolated to exactly the expected
change before regenerating.
Verified: 1095/1095 controller_unit_test (stable across repeated runs),
229/229 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Two related findings logged to .wolf/buglog.json but deliberately not
fixed here (out of scope per discussion): a pre-existing flaky segfault
in the combined controller_unit_test binary, and maxswap/maxtmp being
hardcoded stub values on the worker side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The controller acknowledges EVERY dispatched job through
NotifyData_data_jobDispatchFail (controller/job.c handle_jobdispatch):
a negative code is a real rejection, code 0 is the success ack (the else
branch calls notify_dispatch(jobid, 0)). sched_handle_notify treated any
jobDispatchFail as a failure and returned the job to PEND, so every
successfully dispatched job was immediately re-PENDed, re-elected, and
re-dispatched while the original instance kept running — spawning
duplicate instances cluster-wide. On a 3-worker cluster 24 submitted jobs
ran 38 instances and piled onto the first worker (t1) because its slots
were counted for each duplicate.
Fix: only sched_requeue_pend_job() when fail->code < 0; consume the
success ack (code 0) as a no-op — the job is already RUN
(sched_commit_decision marks it at commit). This was a regression of the
bug-1494 re-PEND logic (bug-1510 fix) that lacked the code guard.
Verified live on t1/t2/t3: 24x sleep-300 jobs distributed exactly 8/8/8,
zero duplicates, all DONE. Add white-box test
test_handle_dispatch_fail_requeues_only_on_negative_code.
Co-Authored-By: Claude <noreply@anthropic.com>
The MCP/apiserver returned the raw 64-bit job ID (e.g. 73930964699447296)
while xce_test/xcecli display the dash form (e.g. 1-0-1786585485-0) that
parse_jobid accepts. A user taking the numeric form into xce_test fails to
parse it. The apiserver is the data-correctness owner (MCP only reads), so
it now emits the controller's fmt_jobid form alongside the numeric ID:
- models: JobSubmitResponse gains job_id_human; job list rows gain
job_id_human (processJobData), using the existing backend.FormatJobID.
- schedulerclient: URMJob.HumanID + SubmitJobResult.JobIDHuman parse it.
- Tests: backend.FormatJobID (real t1 job, pre-cluster1 job, array child);
processJobData adds job_id_human.
Also rename the SSE permission event:subscribe → event:view so mcp_readonly
stays strictly *:view (the read-only invariant test enforces).
Verified live on t1: scheduler_submit returns job_id_human 1-0-1786586474-0
and scheduler_get_job returns human_id.
Co-Authored-By: Claude <noreply@anthropic.com>
Audit of apiserver's three task-4.6 coverage areas (degraded-to-healthy
transitions, concurrent-request safety, candidate-list parity):
degraded-to-healthy and candidate-list parity were already fully
satisfied by tasks 4.1/4.5's own tests. Found one genuine gap: reconnect.go's
needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access
test coverage despite being reached from 5 real concurrent HTTP-wrapper
call sites plus the heartbeat goroutine -- health.go's existing
50-goroutine stress test only covered the separate degraded-until
timestamp. One new test closes it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds an atomic degraded-until state (health.go: IsDegraded/MarkDegraded/
MarkHealthy) that lets request handlers fail fast with a clear
unavailable error instead of blocking on xcecli's own ~100s retry
budget when the backend is known-degraded.
Consolidates what tasks 4.3/4.4 left as duplicated inline
"call xcecli_*; checkAndPerformRejoin()" pairs across five wrapper
functions into one shared callBackend() helper: checks IsDegraded()
before every call (fail fast, skip the C call and the rejoin check
entirely on the degraded path -- checking rejoin there would itself
block up to ~100s, defeating the fail-fast purpose), marks degraded on
-E_XCE_NOT_READY, marks healthy at checkAndPerformRejoin()'s existing
success point (the single call site both the write path and the
idle-period heartbeat path already funnel through).
The circuit breaker's own TTL-based state is deliberately never read
by the rejoin-retry mechanism -- checkAndPerformRejoin() has no
IsDegraded() dependency, so a fast-fail window never suppresses
retry-until-actually-joined; heartbeat.go's unconditional per-tick
call keeps that cadence alive regardless of degraded state, and a
10s TTL (well under the ~100s retry budget) lets write traffic resume
contributing once it clears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
xcecli_heartbeat() is XCECLI_FF_CALL-wrapped -- a deliberately
non-retrying fire-and-forget path that does its own best-effort
one-shot reconnect but never calls xcecli_invoke_rejoin() (that only
fires from the write-call retry path). Left as-is, a controller
failover with no in-flight write request would silently repair the
transport connection without ever re-joining, permanently starving
push notifications.
runHeartbeat() gains two callback hooks fired at the right points --
onFailureToSuccess on a genuine transition, onSuccessTick on every
successful tick thereafter -- wired directly to task 4.3's existing
setNeedsRejoin()/checkAndPerformRejoin() from StartHeartbeat(). No
second rejoin mechanism: checkAndPerformRejoin() is a cheap no-op when
nothing is pending, so calling it unconditionally on every successful
tick is safe and matches the requirement that retry isn't gated on the
transition edge alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Registers a rejoin flag-setter (go_backend_rejoin_handler) as the
client library's reconnect callback: zero I/O, sets an atomic flag,
returns immediately -- avoiding self-deadlock on the library's
per-ctx failover lock the same way worker's and scheduler's rejoin
callbacks already do. The C shim, its extern declaration, and the
xcecli_set_reconnect_handler() registration call all live together in
clib.go's xce_backend_init(), placed after xcecli_start()'s success
check (xcecli_start() wipes any earlier-registered handler as part of
its own init -- the same ordering bug found and fixed in worker's/
scheduler's task 2.3/3.1 work).
A shared checkAndPerformRejoin() helper wraps every gCtx.ctx-based
backend call (SubmitJob, ControlJob, ControlQueue, ControlHost,
FetchFile): after the call returns, if a rejoin is pending, issue
xcecli_join() as a separate step and clear the flag only on success --
on failure it stays set so the next backend call's return retries it,
no separate retry loop.
needsRejoin is a proper sync/atomic flag rather than a plain bool,
since apiserver (unlike the single-threaded C daemons) serves
concurrent HTTP requests that can all reach the shared helper at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Add four tools closing the submit→watch→analyze→suggest loop in the
request/response MCP model:
- scheduler_wait_job: poll a job to a terminal state or timeout.
- scheduler_analyze_failure: gather job record + output logs for failure
analysis (interpretation left to the LLM).
- scheduler_suggest_resources: conservative processor/node suggestions from
current cluster availability.
- scheduler_estimate_wait_time: rough wait estimate from pending count and
free cores (guidance, not a guarantee).
All verified live on t1 against the running apiserver.
Co-Authored-By: Claude <noreply@anthropic.com>
Add WatchJobs(ctx) to the Client interface and HTTPClient: opens the
apiserver's /xce/v1/events SSE stream and yields JobEvent values until the
context is cancelled or the stream ends. JobEvent carries the event type,
job id, status string, exit code, and a raw job snapshot.
Also fix ListJobs' filter params: the apiserver accepts filters via the
"filter" query param in field:op:value form (e.g. queue_name:eq:normal);
the individual queue_name/submit_user params it ignores with a 400, which
broke every filtered list. Multiple conditions are brace-wrapped {a},{b}.
Verified live on t1: WatchJobs captured job.updated PEND→RUN→DONE for a
submitted job.
Co-Authored-By: Claude <noreply@anthropic.com>
Add a Server-Sent Events endpoint /xce/v1/events streaming job lifecycle
events (job.created/updated/deleted/finished) to subscribers. The
DefaultHandler now publishes every job event to a process-wide EventBroker;
the SSE handler subscribes and streams JSON frames with a keep-alive comment
during idle.
- backend/broker.go: thread-safe fan-out Broker (drops for slow consumers so
the notification path never blocks), Event type with job_status string.
- backend/handler.go: OnJob* publish to DefaultBroker.
- controllers/events.go: SSE handler (gin, text/event-stream, keep-alive).
- router + whitelist + rbac: /xce/v1/events route, event:subscribe
permission granted to mcp_readonly/mcp_operator/cluster_admin.
- Tests: broker fan-out/slow-consumer/unsubscribe, handler publishes, SSE
end-to-end via httptest.
Verified live on t1: SSE stream delivered job.updated PEND→RUN events for a
submitted job.
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces apiserver's two connect mechanisms -- a per-peer Go loop that
hardcoded nmasters=1 per attempt, and a separate direct-construction
path for the legacy single-host config -- with one pipeline:
buildClusterConfig() parses (via task 4.1's C parser) or synthesizes a
single-candidate config, validates it, then ConfigConverter
(xce_cluster_config_to_client_config) produces the client config for
one xce_backend_init() call. A 3-candidate config now actually
exercises the C library's own failover/standby-redirect handling on
initial connect, matching worker's and scheduler's task 2.1/3.1 work,
instead of a Go-level per-candidate retry loop that never built a real
multi-master config.
Along the way: found and fixed a genuine use-after-free. The deleted
tryConnect() freed its C.CString-allocated host address immediately
after its single connect call returned, but xcecli_ctx_init() stores a
raw pointer to the config forever (client.c:38) and every later
failover rotation re-dereferences it (client_failover.c) -- so any
reconnect after the first would read freed memory. Fixed by giving the
cluster/client config structs package-level, process-lifetime storage
(gClusterCfg/gClientCfg), mirroring worker's and scheduler's file-scope
static pattern, with host/from/key written into fixed-size C arrays
instead of freed heap pointers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's hand-written Go parsePeer() with a CGO wrapper
around the shared C parser+validator pair (xce_parse_peers then
xce_cluster_config_validate, both src/lib/cluster.c) that worker and
scheduler already use from their own main.c's. Gives apiserver the same
[name=]host:port[,...] syntax (including the name= prefix the old Go
parser could not express) and identical malformed-candidate rejection
(empty host, port <= 0, non-numeric port) -- round-1 review caught the
first pass skipping the validate() half, silently accepting inputs
worker/scheduler both reject.
A malformed entry now rejects the whole candidate list rather than
being skipped with a warning (matches xce_parse_peers()'s own
all-or-nothing construction and worker/scheduler's precedent) --
surfaced loudly via RegisterToBackend's error, not silently.
Parsing only -- RegisterToBackend()'s connect loop and tryConnect()
are unchanged; collapsing them into a single connect call is task
4.2's job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Proves task 3.2's retry-until-success-or-shutdown wrapper's guarantees
hold across multiple queued decisions drained through the real
xce_sched_driver_on_decision(), not just a single isolated call:
decisions queued behind a stuck one publish afterward in original
order (Req 7 AC3), and shutdown mid-retry costs at most one retry
cycle regardless of queue depth -- decisions behind the stuck one see
core.stop already true and are never attempted (Req 7 AC2). Also adds
a source-scan confirming this stays scoped to the scheduler's own
publish direction (Req 7 AC5) and a fresh-instance test confirming a
restart still loses in-flight queued decisions, unchanged by this work
(Req 7 AC6).
Pure test-coverage task -- zero production changes, matching the
review's independently-reproduced RED->GREEN evidence that the
existing task-3.2 implementation already satisfies these guarantees.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds a worst-case-occupancy test proving the scheduler's picker/allocator
pipeline buffers briefly past a stalled publish (decisions cbuf cap 5,
jbuf cap 1) then genuinely blocks until connectivity recovers, with zero
job loss (Req 3 AC2). Adds a source-scan regression guard confirming the
scheduler makes no periodic heartbeat call of its own (Req 3 AC3).
Also fixes a real gap found along the way: sched_init()'s
all-candidates-unreachable and join-failure log lines named only a
candidate count, never which candidates were attempted (same class of
bug worker's task 2.3 fixed). Added sched_format_candidate_list(),
covered by a real dup2()-based stdout capture around a live sched_init()
failure path (round-1 review caught this diagnostic shipping with zero
test coverage; fixed in round 2).
Multi-candidate join and kill-primary/reconnect were validated manually
against a live 3-node cluster rather than committed as an automated
test, following worker's task 2.3 precedent -- no automated live-deploy
E2E harness exists for any C daemon in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Wraps sched_publish_dispatch()/sched_publish_pend_reason() in a shared
retry loop that keeps retrying a -E_XCE_NOT_READY-exhausted publish
until it succeeds or the scheduler is shutting down (core.stop checked
before each attempt, not only after), composed with the pending-rejoin
follow-up established in task 3.1. Extended-retry success logs
distinctly from an ordinary first-attempt success (Req 7 AC4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replace scheduler's inline per-candidate loop and separate single-host
branch with one unified pipeline (mirrors worker's task 2.1 fix): build
exactly one validated candidate list, whether from --peers or synthesized
as a single candidate from the legacy default, then connect through the
shared xce_cluster_config_to_client_config() helper exactly once.
Adds sched_on_reconnect(), a rejoin callback that only sets a flag and
returns immediately (no I/O, no xcecli_* calls) to avoid the self-deadlock
worker hit in task 2.3 — registered via xcecli_set_reconnect_handler()
after xcecli_start(), since xcecli_start() resets any handler registered
earlier back to NULL.
Adds test_scheduler_connect.c: behavioral tests confirm no-peers and
single-candidate configs produce identical connect log sequences and that
unreachable candidates fail fast without retry loops; structural tests
source-scan scheduler.c to confirm a single xcecli_ctx_init() call site, a
single xcecli_join() call site, and a rejoin callback body free of I/O and
xcecli_* calls.
Live-validated against a 3-node HA controller cluster: a scheduler
configured with the standby listed first in --peers fails over via
xcecli's built-in retry-next mechanism to the primary and joins
successfully.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Confirm connector thread is isolated from worker_process_update() (main
thread) and resource_sync_thread() (own thread) via code read plus a real
pthread concurrency test. Add unit and live 3-node E2E coverage for
candidate/reconnect behavior, all-candidates-unreachable diagnostics, and
liveness cadence under concurrent reconnect.
The E2E run surfaced two production bugs the 2.1/2.2 unit tests missed:
xcecli_start() resets the reconnect callback to NULL after it runs, so
registering it before xcecli_start() silently no-ops the whole rejoin
mechanism; and a successful worker join returns its assigned slot index
as reply.status (any rc >= 0), not always exactly 0, so checking_connection()'s
rc == 0 success check never fired on a real non-zero-slot join. Both fixed,
plus per-candidate names added to the all-unreachable failure log.
Root-check bypass needed for the E2E run is gated at compile time via the
existing VIRTUAL_CLUSTER flag, not a runtime env var, so the real xcecd
binary's root requirement has zero bypass surface.
Thread cluster_cfg by value through xcecd_config_t -> xcecd_context_t ->
cc_ctx so it's populated before controller_connector_start() fires the
first join, instead of after xcecd_start() returns. checking_connection()
now retries a failed post-reconnect xcecli_join() on the next liveness
probe instead of leaving needs_rejoin cleared on mere attempt.
