JobSubmitRequest gains a FromHost field, populated in BuildJobRequest()
via os.Hostname(). Matches how traditional batch schedulers report the
submitting host, and sidesteps having the server derive it from the
connection (which apiserver was doing via ctx.ClientIP() -- an IP not
a hostname, and spoofable via X-Forwarded-For since this apiserver
never configures gin's TrustedProxies).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
OnHostConfigUpdated only applied config-sourced fields (MaxSlot,
HostModel, HostType, DisptWindow) onto a host whose name exact-matched
an entry already in the store. hosts.yaml commonly defines only a
wildcard "default" entry, so a dynamically-joined worker never
received its own config push and MaxSlot stayed at Go's zero-value
forever -- decoupled from the C-side scheduling path, which already
resolves the real value via worker.c's wildcard fallback.
The "default" pseudo-host's config is now cached and retroactively
backfilled onto any registered host with no explicit config of its
own, in either arrival order (config-before-worker or
worker-before-config). The -1 HOSTCONFIG_MXJ_AUTO_DETECT sentinel
(bug-1556) is resolved to the host's real NumCPUs when applied via
fallback, mirroring worker.c's own resolution, so the display shows
usable capacity rather than a bare -1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
GetJobsInfo/GetJobByIDs validated the -o/-l field list against
models.JobSubmitRequest{} -- the job-submission payload -- instead of
models.JobInfo{}, the query/response model (the already-correct
pattern used elsewhere in this file at line 1131). Response-only
fields like pending_reason don't exist on JobSubmitRequest, so any
such field made the whole listing fail outright with "unsupported
query field", not just omit that column.
core_limit/mem_limit exist on JobSubmitRequest but not on JobInfo --
switching wholesale would have newly broken those two previously-valid
fields. Added CoreLimit/MemLimit to JobAdditionalFields, the existing
merge-in bucket already used for pids/pgids/interactive_job_url.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
ExtractErrorMessage()'s connection-refused branch returned the bare
OS error string verbatim. A CLI call right after apiserver startup
(before its listener is up) surfaced a raw "get the request failed:
[connection refused]" with no indication it's a transient startup
race, not a real failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds DirtyFingerprint var and composeVersionDisplay() helper to
xcecli/cmd/main.go, a logical duplicate of apiserver's just-approved
implementation (task 4.4, same anchored strings.TrimSuffix strip-and-
append rule). Applied at the sole consumption point: the version string
fed into router.Execute(cm, version), which sets Cobra's rootCmd.Version.
router.go is untouched -- xcecli has no startup log to protect, already
established out of scope by the original build-time-versioning feature.
New main_test.go covers the same 6 cases as apiserver's tests. Verified
independently by review (including a second, independent mutation test)
with real builds across all 3 required scenarios and a full xcecli module
test suite run showing no regressions.
This completes all Go binary work (tasks 4.1-4.5) for this spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DirtyFingerprint package var (default "") and a composeVersionDisplay()
helper implementing the same strip-and-append rule as the C daemons: strips
a trailing -dirty from GitCommit and appends -<fingerprint> when available,
via a genuine anchored strings.TrimSuffix, not a blind substring replace.
GitCommit's declaration and the logger.Info startup-log call site (with its
zap fields) are completely untouched.
New main_test.go covers 6 cases via table-driven tests. Verified independently
by review, including a self-constructed mutation test (distinct from the
implementer's own) proving the anchored-suffix regression test is genuinely
meaningful, a real build across all 5 ldflags scenarios, and a live daemon
startup-log capture confirming no leak into the log even on the same binary
whose -version output shows the fingerprint.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an independent DIRTY_FINGERPRINT computation to src/frontend/Makefile,
mirroring xce-build.sh's task 2.1 algorithm exactly, wired into
VERSION_LDFLAGS as -X main.DirtyFingerprint=. GIT_COMMIT/GIT_STATUS/
BUILD_TIME (including GIT_COMMIT's own -dirty suffix) are completely
untouched.
This took two rejected review rounds to get the failure-detection right:
round 1 found a `{ git diff HEAD; ...; } >file` grouped command whose exit
status only reflected the last command, silently swallowing a git diff
failure and fabricating a hash from incomplete data. Round 2 found the
identical bug class one step further down (git ls-files | sort | xargs
cat's pipe swallowing an ls-files failure). Fixed by pulling git ls-files
out of the pipe into its own directly-redirected, individually-checked
step, deliberately chosen over `set -o pipefail;` (which also worked)
because this repo's own build-deployment.md documents a past incident of
code silently relying on /bin/sh happening to be bash with no SHELL:=bash
pin.
One narrow, explicitly-disclosed residual gap remains (a sort-only
failure, not cat, in the trailing pipe) and was assessed as acceptable
low risk after independent verification across three review rounds.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both apiserver/meson.build and xcecli/meson.build add
XCE_DIRTY_FINGERPRINT=get_option('dirty_fingerprint'), unquoted,
identically positioned to the existing XCE_GIT_COMMIT/XCE_BUILD_TIME
entries.
Verified independently by review with a distinct test value: fresh
configure + real ninja builds of both binaries confirm the value reaches
go build's actual invocation (visible in go version -m's recorded -ldflags
build setting), matches the C-side -DVERSION_DIRTY_FINGERPRINT= from the
same configure, and degrades to empty identically to the two pre-existing
vars when omitted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds GitCommit/BuildTime vars to xcecli's main.go. Cobra's rootCmd is
declared unexported in router.go (a different file than the task's
original main.go-only boundary), so main.go can't reach it directly --
resolved by extending router.go's sole exported entry point,
Execute(cm, version), to accept the composed version string and set
rootCmd.Version before rootCmd.Execute() runs. Cobra then auto-registers
--version/-v natively once rootCmd.Version is non-empty (confirmed via
reading Cobra v1.9.1 source directly) -- zero additional flag-parsing
code, as designed.
Live-verified via both build paths (meson xce-build.sh and make
build-cli): real commit id (matching git rev-parse --short HEAD plus
dirty suffix) and build time, exit 0.
xcecli's startup log deliberately not extended -- Requirement 3 scopes
that to the four daemons only; xcecli is a one-shot CLI. Corrects a
stale design.md line that had assumed otherwise.
Task 4.5/13 -- closes out the Go binary version display phase (tasks
4.1-4.5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Declares GitCommit/BuildTime package vars (default "unknown", target of
task 4.1's -ldflags -X injection), a -version flag checked before
utils.PromptForTopDir()/InitLogger() (zero filesystem side effects), and
extends the existing "Initialize Logger done!" startup log line with
zap.String structured fields carrying both values, per design.md's
explicit mandate.
Live-verified end to end: -version output and the startup log's first
line are byte-identical, with a real commit id (matching
git rev-parse --short HEAD plus dirty suffix) and UTC timestamp -- the
first full round-trip proof in this spec (earlier Go tasks could only
verify via linked build-info metadata, since no consuming vars existed
yet).
Task 4.4/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dev-loop Go build path (no meson involved) now computes commit id (git,
-dirty suffixed, "unknown" on git failure) and build time (UTC date) via
Make := (computed once per invocation), passing the identical -ldflags
shape go_build (task 4.1) already establishes.
Live-verified: make build-cli embeds real, git-matching values (go
version -m). make build-server hits a pre-existing, unrelated CGO
failure (bug-1597, missing CGO_CFLAGS/CGO_LDFLAGS) -- confirmed via
stash A/B that this reproduces identically on the unmodified Makefile,
not a regression from this change. Reproduced apiserver's build
invocation manually with bug-1597's own documented workaround to prove
the ldflags mechanism itself works correctly.
Task 4.3/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apiserver/meson.build and xcecli/meson.build now pass
XCE_GIT_COMMIT/XCE_BUILD_TIME (from the project's commit_id/build_time
meson options) into go_build's environment, mirroring the existing
CURRENT_SOURCE_DIR/BUILD_ROOT env-passing mechanism. Deliberately
unquoted: go_build's fix_input_envs() only quote-strips
CURRENT_SOURCE_DIR/BUILD_ROOT, not these two -- quoting them would embed
literal quote characters and corrupt the -ldflags argument.
Live-verified: real values (matching git rev-parse --short HEAD plus
dirty suffix, and a real UTC timestamp) reach go build's actual
invocation and are embedded in both binaries' Go build-info metadata.
No main.GitCommit/main.BuildTime symbols exist in either binary yet
(confirmed via go tool nm) since apiserver/xcecli's main packages don't
declare those vars until tasks 4.4/4.5 -- expected, this task's scope is
the env-var plumbing only.
Task 4.2/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two retrospective design-review findings:
- design.md's System Flows sequence diagram showed the controller-layer
resval check as an independently-launched client session, implying no
dependency on the injecting worker staying connected. As-built (bug-1579),
host-attrs resources reset to baseline the instant the worker disconnects,
so the check actually runs nested inside the still-open worker session via
xce_test's `!` shell-escape. Updated the diagram and prose to show this,
and documented the still-open residual risk for custom host-attrs
resource names in Error Handling.
- Added explicit "do not change this format" warning comments at both
trace-log call sites (core_events.c's RESVALS_DELTA case, apiserver's
OnResourceLocationUpdated) that xce-test-resources' log-grep depends on --
an unrelated logging cleanup at either site would otherwise silently turn
every future verification run into a false UNREACHABLE/timeout.
No behavior change in either source file -- comments only.
One logger.Debug("resource-location stored", ...) call in
OnResourceLocationUpdated, immediately after store.ResourceLocations.Put,
logging resource/host/value as zap fields. Debug-level only, matches this
package's existing logging convention (reuses convert.go's field names).
Live-confirmed against a real deployment with logging.level: debug.
Also surfaced (but did not fix, out of boundary - pure Go change, no
possible causal link) a pre-existing xce_test segfault on string-typed
resupdate (type/model) - uninitialized stack memory, non-deterministic,
invisible to ASan. Logged as bug-1574; flagged in tasks.md as a risk for
task 7.1's default resource set and task 8.1's live e2e run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.
Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.
Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Audit of apiserver's three task-4.6 coverage areas (degraded-to-healthy
transitions, concurrent-request safety, candidate-list parity):
degraded-to-healthy and candidate-list parity were already fully
satisfied by tasks 4.1/4.5's own tests. Found one genuine gap: reconnect.go's
needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access
test coverage despite being reached from 5 real concurrent HTTP-wrapper
call sites plus the heartbeat goroutine -- health.go's existing
50-goroutine stress test only covered the separate degraded-until
timestamp. One new test closes it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds an atomic degraded-until state (health.go: IsDegraded/MarkDegraded/
MarkHealthy) that lets request handlers fail fast with a clear
unavailable error instead of blocking on xcecli's own ~100s retry
budget when the backend is known-degraded.
Consolidates what tasks 4.3/4.4 left as duplicated inline
"call xcecli_*; checkAndPerformRejoin()" pairs across five wrapper
functions into one shared callBackend() helper: checks IsDegraded()
before every call (fail fast, skip the C call and the rejoin check
entirely on the degraded path -- checking rejoin there would itself
block up to ~100s, defeating the fail-fast purpose), marks degraded on
-E_XCE_NOT_READY, marks healthy at checkAndPerformRejoin()'s existing
success point (the single call site both the write path and the
idle-period heartbeat path already funnel through).
The circuit breaker's own TTL-based state is deliberately never read
by the rejoin-retry mechanism -- checkAndPerformRejoin() has no
IsDegraded() dependency, so a fast-fail window never suppresses
retry-until-actually-joined; heartbeat.go's unconditional per-tick
call keeps that cadence alive regardless of degraded state, and a
10s TTL (well under the ~100s retry budget) lets write traffic resume
contributing once it clears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
xcecli_heartbeat() is XCECLI_FF_CALL-wrapped -- a deliberately
non-retrying fire-and-forget path that does its own best-effort
one-shot reconnect but never calls xcecli_invoke_rejoin() (that only
fires from the write-call retry path). Left as-is, a controller
failover with no in-flight write request would silently repair the
transport connection without ever re-joining, permanently starving
push notifications.
runHeartbeat() gains two callback hooks fired at the right points --
onFailureToSuccess on a genuine transition, onSuccessTick on every
successful tick thereafter -- wired directly to task 4.3's existing
setNeedsRejoin()/checkAndPerformRejoin() from StartHeartbeat(). No
second rejoin mechanism: checkAndPerformRejoin() is a cheap no-op when
nothing is pending, so calling it unconditionally on every successful
tick is safe and matches the requirement that retry isn't gated on the
transition edge alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Registers a rejoin flag-setter (go_backend_rejoin_handler) as the
client library's reconnect callback: zero I/O, sets an atomic flag,
returns immediately -- avoiding self-deadlock on the library's
per-ctx failover lock the same way worker's and scheduler's rejoin
callbacks already do. The C shim, its extern declaration, and the
xcecli_set_reconnect_handler() registration call all live together in
clib.go's xce_backend_init(), placed after xcecli_start()'s success
check (xcecli_start() wipes any earlier-registered handler as part of
its own init -- the same ordering bug found and fixed in worker's/
scheduler's task 2.3/3.1 work).
A shared checkAndPerformRejoin() helper wraps every gCtx.ctx-based
backend call (SubmitJob, ControlJob, ControlQueue, ControlHost,
FetchFile): after the call returns, if a rejoin is pending, issue
xcecli_join() as a separate step and clear the flag only on success --
on failure it stays set so the next backend call's return retries it,
no separate retry loop.
needsRejoin is a proper sync/atomic flag rather than a plain bool,
since apiserver (unlike the single-threaded C daemons) serves
concurrent HTTP requests that can all reach the shared helper at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's two connect mechanisms -- a per-peer Go loop that
hardcoded nmasters=1 per attempt, and a separate direct-construction
path for the legacy single-host config -- with one pipeline:
buildClusterConfig() parses (via task 4.1's C parser) or synthesizes a
single-candidate config, validates it, then ConfigConverter
(xce_cluster_config_to_client_config) produces the client config for
one xce_backend_init() call. A 3-candidate config now actually
exercises the C library's own failover/standby-redirect handling on
initial connect, matching worker's and scheduler's task 2.1/3.1 work,
instead of a Go-level per-candidate retry loop that never built a real
multi-master config.
Along the way: found and fixed a genuine use-after-free. The deleted
tryConnect() freed its C.CString-allocated host address immediately
after its single connect call returned, but xcecli_ctx_init() stores a
raw pointer to the config forever (client.c:38) and every later
failover rotation re-dereferences it (client_failover.c) -- so any
reconnect after the first would read freed memory. Fixed by giving the
cluster/client config structs package-level, process-lifetime storage
(gClusterCfg/gClientCfg), mirroring worker's and scheduler's file-scope
static pattern, with host/from/key written into fixed-size C arrays
instead of freed heap pointers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's hand-written Go parsePeer() with a CGO wrapper
around the shared C parser+validator pair (xce_parse_peers then
xce_cluster_config_validate, both src/lib/cluster.c) that worker and
scheduler already use from their own main.c's. Gives apiserver the same
[name=]host:port[,...] syntax (including the name= prefix the old Go
parser could not express) and identical malformed-candidate rejection
(empty host, port <= 0, non-numeric port) -- round-1 review caught the
first pass skipping the validate() half, silently accepting inputs
worker/scheduler both reject.
A malformed entry now rejects the whole candidate list rather than
being skipped with a warning (matches xce_parse_peers()'s own
all-or-nothing construction and worker/scheduler's precedent) --
surfaced loudly via RegisterToBackend's error, not silently.
Parsing only -- RegisterToBackend()'s connect loop and tryConnect()
are unchanged; collapsing them into a single connect call is task
4.2's job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
rshen/Letmein was a live, checked-in cluster_admin credential in
auth.yaml. Renamed to a non-identifying username with a clearly-marked
placeholder password.
docs: add full ops deployment guide (src/docs/deployment.md)
Covers building and running all four daemons plus xceadmin, based on
direct source verification (CLI flags, config file shapes, TLS posture)
and a live single-node deploy. Documents real production gaps (no
process supervision, plaintext non-controller traffic, exposed dev
creds) rather than treating the repo as a finished runbook.
bug-1463: commit 94f50ce6 (bug-1453 CA-cert-plumbing removal) missed
GetJobOutput's dead tls.Config/loadRootCAs block in request.go, plus
peek.go's SetRootCAs call and serverInfo.CACert field access (the
field no longer exists on APIServerInfo) -- broke the xcecli build.
Removed all three dangling references. Also enabled NewAPIClient's
previously-commented-out InsecureSkipVerify TLS config, since no
xcecli method set any TLS config at all -- any https:// apiserver URL
would have failed cert verification, not just GetJobOutput. Matches
bug-1453's documented design (server-side TLS only, no peer
verification, mTLS deferred as P3 hardening).
Documented both the api-server (http+https) and controller
(server-side-cert-only, no client certs) TLS postures in
.kiro/steering/security.md, replacing a stale line claiming TLS is
handled at the network layer, plus a follow-up note for the planned
HAProxy TLS-termination reverse proxy once its cert type and
apiserver's own listener fate are decided.
controller.capnp declared `struct Data $C.mapname("xce_data_t")` (an
OPCode_get query-result tagged union) at file scope, which shadows
Cap'n Proto's builtin Data blob type for every later `: Data` field in
the same file. This silently miscompiled 7 fields across 6 structs as
a struct-pointer type instead of a raw byte blob: UpgradeConfigRequest
.sha256/.bytes, ReplConfigReply.sha256/.bytes, ReplConfigPushRequest
.sha256/.bytes, ReplEnvelope.payload, SnapshotTransfer's chunk
payload, WalCatchupEntry.payload/.batchPayloads, and CtrlHelloRequest
.key. Production C code had already routed around it for years via
hand-rolled raw capn_ptr manipulation (see request.c's own comments);
nothing before now needed the generated codec/accessors to actually
work for these fields.
Renamed the local struct to XceData (capnp-level name only; the
$C.mapname("xce_data_t") C typedef is unchanged) and explicitly
repointed getdata/pushmsg at XceData by name, since those two fields
correctly want the local tagged union. Updated every derived
generated-identifier reference across 19 hand-written files spanning
controller, lib, scheduler, and the Go apiserver/xcecli CGo bindings.
Fixed 5 call sites in lib/request.c that had been typed against the
old workaround alias but now legitimately receive the correct
capn_data type (a pure type-name swap -- Data_ptr/XceData_ptr and
capn_data share the identical {capn_ptr p} layout).
Verified via full rebuild+test: lib (1/1), controller (3/3, incl. the
job-array push regression tests), scheduler (15/15), Go apiserver +
xcecli (all packages green), and a direct pycapnp round-trip proving
UpgradeConfigRequest.sha256/.bytes now encode/decode correctly.
Bug: bug-1461 (.wolf/buglog.json). Unblocks admin-config-tool task 3.3.
bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
Cross-daemon integration test, test-only, spanning the scheduler (C)
and API-server (Go/CGO) receive paths built in tasks 5.1-5.4/5.3b.
Drives real xce_encode()/xce_decode() wire bytes through the actual
production dispatch chain on both daemons -- extract_data() ->
scheduler_req_handler() -> sched_handle_dataset() ->
sched_on_job_array() and go_push_handler() -> handleDataset() ->
cJobArrayToModels()/dispatchJobArray() -- skipping only vtask_entry()'s
~10 lines of static glue, mirrored verbatim in test code (the
if (callnext) treats-any-nonzero-as-proceed behavior that caused
5.3b's bug is reproduced exactly).
Redelivery is tested via genuinely independent fresh decodes of
separately-encoded bytes, not a decode-once-call-twice shortcut.
Bug-catching capability proven by temporarily reverting task 5.3b's
fix and confirming both the C and Go tests fail meaningfully, then
restoring to green -- confirmed independently by both the implementer
and reviewer.
Task 5.5 of job-array-single-wal-record (Requirements 6.1-6.5, 8.4).
This completes Phase 5 (Compact Scheduler/API-Server Push).
Adds the Go/CGO receive path for Data.jobArray, mirroring task 5.3's
scheduler-side C implementation: rawJobArrayToModels() (pure-Go
base_jobid + i expansion), cJobArrayToModels() (CGO adapter sibling to
cJobToModel), dispatchJobArray() (dedup via the same store.Jobs.Has()
the ordinary Data_data_jobs path already writes through, single
confirmation log only when at least one element was newly
dispatched), and a new case C.Data_data_jobArray in handleDataset.
Elements route through OnJobUpdated, matching the ordinary single-job
push's own routing. The CGO boundary needed no changes -- both
C.xce_job_submit_t and C.Data_data_jobArray were already exposed from
prior tasks, including task 5.3b's extract_data() fix which this
receive path depends on identically to the scheduler's.
store.ArrayJobs (the array-level rollup aggregate) is deliberately
left unpopulated -- confirmed out of this spec's scope, deferred to a
future spec per requirements.md's Boundary Context.
Task 5.4 of job-array-single-wal-record (Requirements 6.2, 6.4, 6.5, 8.4).
When a worker re-joins and already exists in workermgr, makeWorker_()
now overrides the existing slot instead of returning EEXIST. Admin-set
flags (LOCKEDU, CLOSED, HIDDEN, CLOSED_BY_ADMIN) survive the override.
Add XCE_WORKER_CLOSED_BY_ADMIN flag to distinguish admin-closed hosts
from workload-closed ones. worker_ctrl sets/clears it alongside CLOSED.
Refactor worker_init/worker_destroy into copy_worker/kill_worker helpers
shared by both fresh creation and override paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three fixes:
1. ExecHosts.UnmarshalJSON: JSON null (sent when no allocation exists)
was treated as a non-"-" string because json.Unmarshal(null, &str)
returns nil with str="". This created a phantom HostSlot{NumSlots:1,
HostName:""} producing "<1*>" for every PEND job. Now null and ""
are both treated as empty hosts.
2. page_print_jobs.go: added !startTime.IsZero() guard so a zero
timestamp ("0001-01-01T00:00:00Z") never produces a start line
even if the status check is bypassed.
3. Added PENDING_REASON field to JobFieldsMap so it is requested from
the API when -l is passed, and rendered in printJobsVerboseOutput
for PEND/PSUSP jobs before the start-time section.
- parsePeer: use net.SplitHostPort (handles IPv6, stdlib convention)
- tryConnect: take Go strings, own CGO allocation internally
- RegisterToBackend: remove unreachable 'no HA candidates' error path
- Sync Peers field comment between utils and backend config structs
