collect_job_rusages() (xcecd_cgroups.c) reads CPUUsageNSec/MemoryCurrent/
MemorySwapCurrent/TasksCurrent via D-Bus and only checked whether the
round trip itself succeeded (r >= 0). systemd's own convention -- not
previously accounted for here -- is to report UINT64_MAX for a counter
that isn't actually being tracked yet, not an error and not zero.
For MemoryCurrent specifically, an unguarded sentinel read produced
(float)UINT64_MAX / (1024*1024) == 17592186044416.0 (2^44) as a single
job's "memory usage", which the scheduler's rusage reconciliation then
summed across every RUN job that hit the same condition -- a live replay
session showed exactly 19 * 2^44 = 334251534843904.0 ("used=304.00 TB")
for 19 affected jobs, matching to the bit.
Added sd_resource_counter_is_available() and gated all four property
reads on it. Applied uniformly (not just MemoryCurrent) since systemd
uses the identical convention for the other three counters. Left the
existing memory_rss MB wire unit alone (bug-1735's established
contract, also relied on by the controller's own display pipeline) --
changing it would need auditing that separate consumer, out of scope
here.
bug-1814.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1554: init_host_attrs() replaces hardcoded placeholder stubs with real
detection -- maxswap from sysinfo()'s totalswap (reusing the same syscall
snapshot already taken for maxmem), maxtmp from a fresh statvfs("/tmp")
total-capacity measurement. No synthetic fallback on detection failure, so
a system with swap genuinely disabled correctly reports 0.
bug-1776: the scheduler simulator's "info worker" assigned-jobs listing now
annotates a job consuming more than one "slots" unit (e.g. "jobid*2"),
using per-host allocation data that was already being walked but discarded.
Falls back to the pre-existing per-job listing when jobs on the same worker
have differing slot counts, since a single suffix on a compressed range
would be ambiguous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
Adds white-box tests for job_start_remote_task()'s launch-type gate
(MPI types defer to their own remote-launch mechanism; default type
fans out via xcecli_runtask() across every allocated host, verified
via a #define-before-#include mock capturing taskrun/taskhost
contents) and create_mpi_cmdline()'s host-list/command construction
for openmpi and mpich. Zero production code changes -- this closes a
previously zero-coverage gap in existing, unmodified worker code
(Requirement 4.4), independent of this feature's resreq wiring work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Extends task 4.3's post-join gating to resource_sync_thread()'s
independent periodic reporting pthread, closing the Requirement 3.8 gap
where load-indices sends could interleave with an in-flight job-state
reconciliation report. Also fixes two pre-existing latent bugs
(bug-1756, bug-1757) in resource_sync.c that blocked a truthful RED/GREEN
signal for the new tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ypUeKgbAqLWUkaSJ79poN
Arm a reconciliation round in apply_join_state() the moment a worker
host becomes connected (first join, reconnect, or failover-reconnect -
all routed through the same shared call site, Req 1.1-1.3), then send
the query from on_worker_join() strictly after the join reply and
existing config push (Req 2.1-2.2), verified via a structural test
that reads real wire order off the endpoint queue.
Also closes Req 3.7's config-delivery seam end-to-end: worker_resync_
window_ms now rides the existing Conf4Worker push (clustername
precedent, no new opcode) and the worker applies it to its retention
window on receipt.
Fixes bug-1752: myreq_handler had no OPCode_push dispatch case at all,
so every config push - including the pre-existing clustername field -
was silently discarded. Confirmed harmless until now (clustername was
never read anywhere in src/worker/); closed narrowly for
worker_resync_window_ms only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add OPCode_jobStateQuery dispatch to myreq_handler with multi-page query
reassembly, routing to task 4.2's xcecd_resync_answer_query() (Req
8.1, 8.3). Gate the worker's resumption of normal post-join operation
on a resync_pending flag until the full report has sent (Req 3.8).
Fixes bug-1749 (xcecd_resync.c was never added to the worker's
production source lists - task 4.2's build only ever exercised it via
the test binary's white-box include).
Round 1 review caught bug-1750: the gate sat as checking_connection()'s
entry guard, so a connection drop mid multi-page query left
resync_pending stuck forever, permanently disabling the worker's only
reconnect mechanism (Req 6.1 violation). Round 2 narrows the gate to
guard only the rejoin action, and resets it from xcecd_on_reconnect()
so a stalled query can never wedge it past one reconnect cycle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Classify each queried job ID as running/finished-ok/finished-fail/
unrecognized (Req 3.1-3.3, 3.6), report unqueried running/recently-
finished jobs as extras, and send both via task 1.4's paginated wire
helper (Req 8.2, 8.3). Composes correctly with the already-implemented
controller-side consumer (resync_apply_report).
Adds xcecd_finished_job_retention_snapshot() to xcecd_worker_utils, a
small additive companion needed to enumerate all retained jobs for the
extras set (task 4.1 only exposed single-jobid lookup).
Folds the still-open OPCode_jobStateQuery receive-dispatch wiring gap
into task 4.3's scope in tasks.md, since 4.3 already needs the real
receive path to implement its post-join gate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Retain a job's exit status/code/true finish time for a configurable
window after update_worker_status() detects its exit, before
cleanup_worker_list() would otherwise discard all trace of it
(Requirements 3.4, 3.5, 3.7). Hooked into the single choke point both
production exit-detection paths (libuv IPC callback, dbus polling)
converge through, so retention happens-before any reap regardless of
which mechanism is active.
Also prunes on retain (not just on lookup), closing a review finding:
lookup currently has zero production callers until task 4.2 lands, so
without this the store would grow unboundedly for the daemon's uptime.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
A JOB_EXITED status could mean either a deliberate, expected termination
(worker_signal_callback() only ever sends SIGTERM/SIGKILL to cancel/
force-kill a job) or a genuine unexpected crash signal. Both logged at
ERROR, drowning real crashes in expected-cancellation noise. Added
job_exit_log_level(): exit_code == 128+SIGTERM/SIGKILL logs at INFO
(matching process_exit_callback()'s existing plain-INFO treatment for
the same case), anything else stays ERROR. Verified: worker test suite
5/5 including the new test_uv_ipc_job_exit_severity coverage.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Extends bug-1588's job.c fix to the remaining raw-%lu jobid logging sites:
8 worker daemon files (xcecd_worker.c, controller_connector.c, uv_ipc.c,
xcecd_worker_utils.c, xcecd_dbus_manager.c, xcecd.c, main.c, xcecd_cgroups.c)
and src/controller/testsuite/test.c (xce_test, heavily used for controller
testing). 99 call sites converted to fmt_jobid()+%s, matching the existing
convention. Non-jobid %lu values (lsn, client id, byte offset) and raw-numeric
identifier/path construction (checkpoint dir, systemd unit name, XCE_JOBID
env var) are deliberately left untouched.
Verified via /xce-build worker and controller (asan, --incremental): both
compile clean with zero warnings tied to the change; worker suite 4/4 OK,
controller suite 4/5 OK (the one failure is the pre-existing bug-1561
overlayfs fdatasync flake, unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker's hand-rolled OPT_VERSION case now composes via a genuinely
separate version_display_buf/xce_compose_version_ex() call, mirroring
task 3.2's already-approved controller/scheduler pattern exactly. Both
startup-log XCECD_LOG call sites (--peers and no-peers) are left
completely untouched, same xce_compose_version() call, same version_buf.
Verified independently by review with a real build and live daemon runs
in both branches: --version shows <commit>-<fingerprint> matching
compile_commands.json, -dirty correctly stripped not doubled; both
startup log branches show the old fingerprint-less format unchanged.
All three C daemons (controller, scheduler, worker) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker uses plain getopt_long(), not argp, so it has no free --version
handling like controller/scheduler. Adds a long-only OPT_VERSION=257
option and a handling case that computes xce_compose_version() into a
static buffer and returns before any startup work (check_user()/
xcecd_init/xcecd_start). Both startup INFO log branches (--peers and
no-peers) extended to log the identical buffer.
Live-verified: --version and both startup-log branches are
byte-identical (captured via a real daemon run under sudo).
Also updates bug-1519 (pre-existing, unrelated ASAN "Joining already
joined thread" abort during worker shutdown): independently reproduced
under real root during this task's review, refining its prior "never
reproduced under real root" note -- not a userns/unshare artifact, still
unfixed, does not affect this task's code path.
Task 3.3/13 of build-time-versioning. This closes out the "Core: C
daemon version display" phase (tasks 3.1-3.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every early-exit branch in the worker_list scan either re-tested the same
node forever (bare `continue`) or returned out of the whole function on the
first DONE/EXITED worker, so job array elements past the head of the list
never got their on_job_start report. Capture `next` before touching each
node and always advance to it. Affects xcecd_virtual and real xcecd alike
(same DBUS_MANAGER-gated function).
Also records a roadmap follow-up (xcecd_virtual concurrent-job support,
MXJ > 1) discovered while investigating this bug.
collect_load_indices() (resource_sync.c) tagged every periodic
mem/r1m/r15m/r15s/ut/pg/ls/it/swp/io/tmp report with gethostname()'s
raw OS/container hostname instead of the worker's own join identity
(xcecd_ctx.virtual_name, the -n/--name value). Every other worker-keyed
lookup downstream (scheduler's sched_get_worker(), and the same
HOOK_DETAIL_RESOURCE_UPDATE payload forwarded on to the Go apiserver)
matches by that join identity, so the mismatch silently dropped every
periodic resource report for the real worker.
Also adds a startup INFO log (main.c) naming the resolved job picker
plugin, used to rule out a legacy-picker hypothesis while chasing a
related (still-open) dispatch issue.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirm connector thread is isolated from worker_process_update() (main
thread) and resource_sync_thread() (own thread) via code read plus a real
pthread concurrency test. Add unit and live 3-node E2E coverage for
candidate/reconnect behavior, all-candidates-unreachable diagnostics, and
liveness cadence under concurrent reconnect.
The E2E run surfaced two production bugs the 2.1/2.2 unit tests missed:
xcecli_start() resets the reconnect callback to NULL after it runs, so
registering it before xcecli_start() silently no-ops the whole rejoin
mechanism; and a successful worker join returns its assigned slot index
as reply.status (any rc >= 0), not always exactly 0, so checking_connection()'s
rc == 0 success check never fired on a real non-zero-slot join. Both fixed,
plus per-candidate names added to the all-unreachable failure log.
Root-check bypass needed for the E2E run is gated at compile time via the
existing VIRTUAL_CLUSTER flag, not a runtime env var, so the real xcecd
binary's root requirement has zero bypass surface.
Thread cluster_cfg by value through xcecd_config_t -> xcecd_context_t ->
cc_ctx so it's populated before controller_connector_start() fires the
first join, instead of after xcecd_start() returns. checking_connection()
now retries a failed post-reconnect xcecli_join() on the next liveness
probe instead of leaving needs_rejoin cleared on mere attempt.
Replace worker's separate default/single-host and multi-candidate connect
branches with one path: synthesize-or-accept a candidate list, validate,
convert via xce_cluster_config_to_client_config(), connect once. Register
a flag-only rejoin callback (no I/O, no client-lib reentry) and remove the
liveness probe's bounded give-up so the library owns retry/recovery.
N_MAX_MASTERS (client.h) now derives from config.h's MAX_HA_CANDIDATES
instead of an independently-maintained literal (3 vs 4 previously).
Fixes worker's and scheduler's hardcoded `> 3` startup checks to
reference the constant symbolically, and updates the client-retry
test fixture so it doesn't exercise an uninitialized 4th slot.
Task 1.1 of component-failover-adoption (Requirement 1 AC7).
bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
Three identical static copies (controller, worker, scheduler) replaced
with a single xce_parse_peers() in src/lib/cluster.c.
The libxce version is the canonical implementation: stores the logical
name from 'name=host:port' syntax so daemons on the same dev host can
identify themselves by name rather than IP address.
Worker (xcecd):
- Add xce_cluster_config_t cluster_cfg field to controller_connector_ctx_t
(controller_connector.h includes xce/cluster.h)
- Populate cc_ctx->cluster_cfg from parsed cluster_cfg after xcecd_start()
sets cc_ctx (main.c, removes the P3 placeholder)
- Replace xcecd_join_cluster() single-shot connect with a candidate-iterating
loop: tries each candidate, calls xcecli_stop() before the next attempt,
falls back to the original single-host path when candidates_count == 0
Scheduler (xce_scheduler):
- Add xce_cluster_config_t cluster_cfg field to xce_scheduler_t
(scheduler.h includes xce/cluster.h)
- Add sched_set_cluster_cfg() setter
- Replace the hardcoded xcecli_start/xcecli_join pair in sched_init() with
the same candidate-iterating pattern; original single-host path preserved
- main.c calls sched_set_cluster_cfg() before sched_init() when --peers given
(removes the P3 placeholder comment)
checking_connection() already calls xcecd_join_cluster() on retry, giving
the worker free reconnect-after-failover behaviour for no additional code.
Worker (P2.5-5):
- Added parse_peers() static helper to src/worker/src/main.c
- Switched getopt to getopt_long to support --peers and --ca-cert long options
- When --peers is given: parses into xce_cluster_config_t, calls
xce_cluster_config_validate(), logs INFO with candidate count/identity/ca_cert
- If validation fails: prints clear error and exits(1)
- WARN logged when candidates_count > 3
- Backward compatible: no --peers falls back to existing single-controller path
Scheduler (P2.5-6):
- Added parse_peers() static helper to src/scheduler/src/main.c
- Extended argp struct with --peers (-P) and --ca-cert (-C) options
- Same validation/logging pattern as worker
- Backward compatible: no --peers falls back to existing single-controller path
Tests (new):
- src/lib/testsuite/test_cluster_parse.c: 8 unit tests for parse_peers logic
(single entry, multiple entries, no-name format, empty/null/missing-port errors,
name-stripped correctly, MAX_HA_CANDIDATES cap respected)
- Registered in test.c and meson.build
- All 8 tests GREEN; 3 pre-existing failures in unrelated GC/CSE tests unchanged
