xcecli_jobsubmit() could return rc>=0 (success) while leaving its *jobid
output parameter at 0, reproducible under concurrent client load
immediately after a fresh single-node election (~1-3/9 clients per round
in the original repro).
Root cause: XCECLI_FAILOVER_CALL's idempotent-replay rule
(client_failover.c's xce_failover_classify) normalizes a retried resend
that hits the server's EEXIST/inflight-collision rejection into overall
success -- but that rejection reply carries no payload
(handle_jobsubmit's EEXIST branches reply with RequestReply_data_nodata,
job.c), so xce_jobsubmit() never populates *jobid on the winning
attempt, nor did it on the earlier failing attempt (raw status was
already negative).
Adds jobsubmit_finalize_jobid() (client_internal.h/client.c), called
from xcecli_jobsubmit() right after XCECLI_FAILOVER_CALL returns. Sets
*jobid = submit->jobid (the client-generated id already on the wire --
the controller commits/echoes this exact value on every genuine success
reply, bug-786/job.c:1439) whenever the overall call succeeded,
otherwise leaves it untouched. Placed at the jobsubmit-specific wrapper
layer, not the generic classify/decide layer, which stays fully generic.
3 new socket-free unit tests in test_libxce_retry.c (following this
codebase's direct-state-simulation style for I/O-adjacent logic).
Live-validated: full ASAN rebuild clean, libxce test suite 1/1 OK with
all 3 new tests passing, and a 5-round/45-client concurrent-submit
repro against a live single-node cluster showed 0 "no clean ack"
failures (vs. 1-3/9 per round pre-fix).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds `xceadmin config dump CLUSTER_BIN_PATH`, the reverse of `config
build` -- decodes a serialized ClusterConfig back to human-readable text
so a built config can be inspected without a throwaway pycapnp script.
Motivated directly by how bug-1556 (hosts.yaml's max_slot: -1 auto-detect
sentinel silently becoming mxj=0) was actually found: by hand-decoding a
live cluster.bin. The dump renders the same sentinel as
"max_slot=-1 (auto-detect from CPU count)", not the raw uint32.
Also adds parse_size_or_number() to xcecommon.mapping so a queue/host's
sched_stop threshold on a byte-valued resource (maxmem/mem/tmp/maxswap/
maxtmp) can be written as a human-friendly size string ("500MB", "2GB")
instead of a raw byte count; a plain number still passes through
unchanged for non-byte resources like load averages. hosts.schema.json
and queues.schema.json's sched_stop schema gains a matching SizeOrNumber
type. dump_cluster_config() humanizes the same byte-valued thresholds
back on the way out (_format_bytes_human).
New ClusterBinFileError covers every local cluster.bin read/decode
failure mode without letting a raw OSError/capnp.KjException escape the
CLI. RESOURCE_VALTYPE/RESOURCE_ORDTYPE/RESOURCE_FLAG_* made public in
mapping.py so dump_cluster_config can reuse the same int<->name tables
build_cluster_config already uses, keeping both directions in sync by
construction.
256 xceadmin tests pass (incl. new test_config_dump.py and the
parse_size_or_number/SizeOrNumber cases in test_mapping.py/test_cli.py),
verified via `xce-build.sh xceadmin`'s full pinned out-of-tree install
+ pytest run before this commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.
Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.
Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apiserver now returns pending_reason as an object ({main_reason,
main_reason_name, host_reasons, res_reasons}) since the PEND-reason work.
xcecli's QueryJob.PendingReason was a plain string, so every job list
failed with "cannot unmarshal object into string". Two fixes:
- xcecli/models/jobs.go: PendingReason is now a string-backed type with a
custom UnmarshalJSON that accepts null / empty string / plain string
(older API) / the object form (current API, rendered as readable text —
LSF bjobs -p style: "match_fail; host t1: match_fail; ..."). Unknown
objects fall back to the raw bytes instead of erroring, so a job list
can never fail on an unexpected pending_reason shape.
- apiserver/models/jobs.go: JobAdditionalFields gains PendingReason so the
fields= query (xcecli `jobs list -l`) no longer rejects pending_reason
with "unsupported query field".
- xcecli/common/output/page_print_jobs.go: adapt the verbose printer to the
new type.
Tests: xcecli/models/jobs_test.go — 6 cases (null / empty / plain string /
object / within QueryJob / unknown object). xcecli 9 packages + apiserver
models/controllers/utils all pass.
Verified on the test cluster: `xcecli jobs list` (no unmarshal error) and
`xcecli jobs list -l` show "PENDING REASONS: host_unavail; host t1:
host_unavail; host t2: match_fail; host t3: match_fail".
Co-Authored-By: Claude <noreply@anthropic.com>
Covers the case where the worker process stays alive and only the TCP link
to the controller drops (or the controller fails over): libxce's failover
loop reconnects transparently and xcecli_hello() returns 0, so
checking_connection() sees ret==0 and never takes its
xcecd_join_cluster()/resend_job_status() path — job state changes that
occurred while disconnected are never re-reported, leaving stale RUN/UNKWN
jobs on the controller.
- worker/controller_connector.c: new xcecd_reconnect_cb registered via
xcecli_set_reconnect_handler after a successful join, resending the actual
job state (bug-1509 semantics) on every transparent reconnect.
- lib/client.c (bug-1522): xcecli_start no longer resets rejoin_cb/rejoin_arg
to NULL — it is initialized by xcecli_ctx_init's calloc, and a pre-start
registration (scheduler sched_reconnect_cb, worker xcecd_reconnect_cb) must
survive first connect; the failover loop invokes it after a transparent
reconnect (xce_ctx_reconnect), which never re-runs xcecli_start. Previously
every pre-start registration was dead code.
- scheduler.c: move xcecli_set_reconnect_handler to AFTER a successful join —
xcecli_join also runs the failover loop, and a candidate-iteration rotate
(standby redirect) must not fire the rejoin hook before the session exists
(was firing sched_reconnect_cb during initial join → rejoin on
uninitialized state → invalid-opcode crash).
- test_libxce_retry.c: regression test for bug-1522
(test_reconnect_handler_survives_start_state_reset). Suite 296/296.
Functional verification on the test cluster: worker joins (host OK), runs a
job, and after a controller-link break re-joins ("Rejoined cluster
successfully after retry N"). Scheduler stays up (no invalid-opcode crash).
Co-Authored-By: Claude <noreply@anthropic.com>
Add queue dispatch-window support so a queue whose window is closed
(queues.yaml dispatch_window, e.g. 19:00-08:30) is treated as
non-electable and reports PREASON_TIME_WINDOW.
- lib/window.c: new xce_timewindow_is_open() — NULL/empty always open,
same-day and overnight (close < open) segments, day-scoped (0=every day,
1-7=Mon..Sun).
- scheduler: make_sched_queue duplicates dispatchwindow_config onto
xce_sched_queue_t.dispatchwindow (freed by kill_sched_queue);
sched_queue_is_ready() checks it; sched_queue_status_reason() now takes
the queue (was a bare status bitmask) and maps a closed window to
PREASON_TIME_WINDOW.
- fcfs: keep the windowed-out queue skip; do NOT attach the queue-status
reason to job->schedobj from the picker thread (bug-1521: races the
allocator thread's MOVE_PTR(obj->reason,...) and corrupts the heap).
- tests: test_window.c (5 cases) covers no-window/daily/overnight/
day-scoped/multi-seg; full suite 295/295 pass.
Cluster-verified: with a 09:00-17:00 window on the xce queue (currently
outside the window), fcfs reports "queue xce ... ready=0 reason=8"
(PREASON_TIME_WINDOW).
Co-Authored-By: Claude <noreply@anthropic.com>
Two bugs surfaced by the new scheduler_pend_reasons tool:
1. The apiserver returns job_status as the numeric XCE state bitmask
(PEND=0x01, RUN=0x02, EXIT=0x04, DONE=0x08), but getJobs read it with
rowStr (string), which returns "" for a number → every job mapped to URM
"unknown", so the pending/running status filter matched nothing.
→ JobStatusFromXCEState(int) + rowURMStatus fallback.
2. pending_reason is a JSON object in the row, not a string; rowStr returned
"" so Extensions["pending_reason"] was empty and the reason never parsed.
→ rowJSON(key) stores the raw JSON value; the MCP pend_reasons parser
handles the map form.
schedulerclient tests updated (JobStatusFromXCEState table). mcp-server +
schedulerclient test suites pass; live MCP call now returns the bjobs -p style
explanation for a PEND job.
Co-Authored-By: Claude <noreply@anthropic.com>
New read-only tool that answers "why is my job waiting": lists PEND jobs (or
one by id) and translates each job's pending_reason into a human-readable
scenario (LSF bjobs -p style). Maps the PREASON_* codes surfaced by the
PEND-reason work (4e2ac86): no_resource / match_fail / limit_global /
limit_host / queue_closed / queue_draining / time_window / host_unavail.
- internal/pendreason.go: pending_reason JSON parsing (mirrors apiserver
rawPendReason) + reasonScenario() translation + bjobs -p style summary.
- internal/app.go: HandlePendReasons — id param for one job, else pending
list filtered by queue/owner/limit.
- cmd/main.go: register scheduler_pend_reasons (read-only).
- 5 unit tests (known/unknown codes, parse from JSON, missing reason,
summary format). go test ./... + go vet clean.
Co-Authored-By: Claude <noreply@anthropic.com>
P0 (done, cluster-verified):
- main_reason was always 0: add_job_preason only set it when BOTH host and res
were NULL, but the allocator always attaches host/res detail rows. Now the
first detail add wins as main_reason, so the API's main_reason_name is
meaningful (was "unknown(0)"). Verified live: PEND jobs now report
main_reason=3/match_fail + per-host reason_name.
- limit plugin attach: host-based quota hit (hg1_slots_per_host) -> LIMIT_HOST
+ limit name; global quota hit (ug1_slots, ...) -> LIMIT_GLOBAL + limit name.
Previously both blocks were silent (job just stayed PEND).
- Extended the PREASON_* enum (was 3 codes) with LSF-style classes:
LIMIT_GLOBAL/LIMIT_HOST/QUEUE_CLOSED/QUEUE_DRAINING/TIME_WINDOW/HOST_UNAVAIL.
apiserver pendReasonName maps them; host/res detail rows now carry reason_name.
P1 (implemented):
- queue status: sched_queue_is_ready no longer a stub — closed/inactivated
queues are not electable; the fcfs election attaches QUEUE_CLOSED /
QUEUE_DRAINING to every PEND job in a skipped queue.
- worker filter: sched_try_alloc_on_host_ skips UNAVAIL/CLOSED workers and
attaches HOST_UNAVAIL instead of trying to allocate on them.
Unit tests: preason suite updated for the main_reason change + 3 new tests
(limit host/global, first-add-wins); 295 lib tests pass under ASAN.
apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
The apiserver stayed circuit-breaker-degraded after a full controller
restart until manually restarted. Two root causes:
1. Breaker recovery was decoupled from the breaker itself. The breaker can
be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while
the controller is electing) with no heartbeat ever failing, but
StartHeartbeat only recovered on a fail→success heartbeat transition
(prevFailed), so the breaker stayed degraded forever.
→ extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when
prevFailed OR the breaker is currently tripped. 5 new unit tests.
2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's
internal ctx state is XCE_CTX_ERR. After a controller restart whose
election outlasts the FAILOVER_CALL retry budget, the retry loop's
xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the
FF path never reconnects and the client stays disconnected forever.
→ lib: new xcecli_force_reconnect() forces a rotate+recover regardless
of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while
degraded.
Verified on cluster: after controller kill+restart, the apiserver logs
"backend re-joined controller after reconnect" and job submissions resume
without a manual restart; a fresh job runs to DONE. Controller, scheduler
(rejoin ok, bug-1513) and workers all auto-recover.
Co-Authored-By: Claude <noreply@anthropic.com>
The "libxce reconnect heap corruption" (ASAN bad-free, free(): invalid
pointer) was NOT a libxce bug — xce_ctx_reconnect is fine. The crashes are
in the scheduler's rejoin state re-sync path (config/dataset re-push),
fixed with the reconnect/rejoin feature now integrated:
1. group.c makeGroup refresh: FREEUP(m->u.name) raw-freed shared interned
xstrings (makeString is a global refcounted table); the correct release
is dereference(), matching group_killer. → dereference().
2. limit.c limit_update_config: after key_add returns -EEXIST (rejoin
re-push — limit name already present) it called kill_limit(limit), but
xkvmgr_add's vclear already freed the limit via the kvmgr datakiller
(=kill_limit). → skip on -EEXIST (idempotent re-push).
3. handler.c/scheduler.c: the config push was applied inline on the vcpu
thread (sched_init_config adds queues/plugin queues) while job updates
flow through pend_updates on the main loop — concurrent rejoin re-push
let fcfs_update_job see a NULL j->queue (SIGSEGV). Route the config
push through pend_updates (XCE_SCHED_OP_UPDATE_CONFIG, killer case
already existed) and process config requests FIRST in sched_update via
a staging queue so jobs always see their queues/resources.
Verified: 295 lib unit tests pass under ASAN (incl. 2 new group-refresh
regression tests), scheduler survives 3 consecutive controller restarts
(each rejoin ok), full job lifecycle (submit→dispatch→DONE) on the
cluster. Also fixed stale build deps (libxce_deps missing librtti_dep,
scheduler_deps missing ssl_dep/c_capnproto_dep, unity include path).
Co-Authored-By: Claude <noreply@anthropic.com>
component-failover-adoption Req 4: when the apiserver's controller connection
is down, backend-requiring API requests must return a clear service-unavailable
error rather than hanging. Adds:
- health.go: an atomic circuit breaker (IsDegraded/MarkDegraded/MarkHealthy).
- callBackend: the four write calls (jobsubmit/jobctrl/queuectrl/hostctrl)
now route through it — degraded ⇒ immediate "backend controller unavailable"
error without invoking the lib; a transport OR XCE not-ready/masterless/
standby failure (E_XCE_NOMASTER/REDIRECT/NOT_READY/STANDBY) trips the breaker
so the NEXT call fails fast. Semantic rejections (-EINVAL/-ENOENT/E_XCE_BAD_*)
never trip it.
- StartHeartbeat: a failed tick marks degraded; recovery re-joins (rejoinBackend)
and marks healthy.
Verified live (t1 cluster): kill controller → first submit returns
xcecli_jobsubmit failed: -10013 (bounded, no hang), second submit returns
"backend controller unavailable (reconnecting)" in ~11ms. 7 unit tests.
Recovery-via-heartbeat after a FULL controller death is NOT yet reliable — the
lib transport reconnect does not always re-establish to the restarted
controller (same class as the scheduler reconnect; tracked under task 13).
Co-Authored-By: Claude <noreply@anthropic.com>
Deep-dive: the rejoin works but crashes ~3-8s later with a REAL heap corruption
(ASAN bad-free freeing a pointer 56B into a 60B region, all-loader frames;
non-ASAN debug build aborts with free(): invalid pointer, gdb stack corrupted).
The endpoint-unmonitor fix (correct, matches xce_ctx_stop) did not help.
Candidate roots: config re-push re-running sched_init_config non-idempotently
(sharedInst MOVE_PTR/zarray_put), lib reconnect vs in-flight vtask, or
state-resync merge. All scheduler reconnect changes reverted to keep the
scheduler stable; this is a dedicated deep-debugging task.
Co-Authored-By: Claude <noreply@anthropic.com>
Closes the G-5 test gap mirroring the G-8 routing test: the rusage notify
handler enqueues XCE_SCHED_OP_JOB_RUSAGE and sched_update() must dispatch it to
sched_apply_job_rusage(), storing the report on the matching job (bug-765 G-5
remediation, production fix already in 93124c5). Driving the real sched_update()
dequeue loop with a recording stub, asserts the request reaches
sched_apply_job_rusage and the job's rusages field is set.
11/11 sched_redispatch_test PASS; full scheduler suite 6/6 OK. Live: job ran to
DONE on the deployed scheduler (G-3 leak fix in place), zero ASAN errors.
Co-Authored-By: Claude <noreply@anthropic.com>
component-failover-adoption Req 2 (partial): two of the three daemons now
self-heal after a controller restart without an operator restart.
- worker (controller_connector.c): checking_connection() previously gave up
after 5 retries and called handle_disconnect() (which set running=false and
exited the connector thread). It now retries indefinitely — the connector
thread is dedicated to this loop so the unbounded retry/sleep cannot block
the worker's other liveness duties. Verified live: killing the controller
leaves all 3 workers alive and retrying; they re-join when it returns.
- apiserver (clib.go): xcecli_heartbeat is fire-and-forget — it best-effort
reconnects the transport but never re-joins, so after a controller restart
push notifications stop. StartHeartbeat now tracks heartbeat failure→success
and, on recovery, calls xce_backend_rejoin (xcecli_join as HostRole_apiserver)
to re-register and resume pushes. Verified live: "backend re-joined
controller after reconnect" logged during a controller kill/restart.
Scheduler reconnect NOT included: the rejoin works but exposes a pre-existing
libxce race — xce_ctx_reconnect's endpoint_fini tears down the old endpoint
while a lib vtask/io thread is still processing it, a deterministic double-free
(bad-free in the loader, same freed address offset across runs). Root-cause in
libxce netio/endpoint lifecycle is a separate task (documented in
left_scheduler_work.md).
Co-Authored-By: Claude <noreply@anthropic.com>
Closes the msg-gap G-8 test gap the research flagged ("handler→dequeue ROUTING
is verified by code review, NOT an automated integration test"). The
remediation (UPDATE_JOB_FIELDS op routed to sched_update_existing_job) is
already in the code (93124c5); these tests lock the routing:
- test_update_fields_routes_to_inplace_path: driving the real sched_update()
dequeue loop, an UPDATE_JOB_FIELDS request reaches sched_update_existing_job
and mutates the existing job in place (same object, runtime state kept).
- test_bulk_update_existing_job_no_damage: a re-pushed dataset row for an
existing jobid is EEXIST-dropped by key_add (via the real sched_add_job) —
the live job keeps identity/state/runtime state.
10/10 sched_redispatch_test PASS; full scheduler suite 6/6 OK. Live cluster:
job PEND→RUN→DONE via notify updates, no WARN, no duplicate instance.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the PEND reason taxonomy (docs/pend-reason-investigation-design.md) to
left_scheduler_work 八.4 推迟/未决, with the three open decisions for unified
analysis: code strategy (reuse 3 codes + subcodes vs expand PREASON_* to ~20),
wiring priority (P0 main_reason + limit attach), and bjobs -p-equivalent MCP
capability.
Co-Authored-By: Claude <noreply@anthropic.com>
Survey of every mechanism that can block a job from dispatch across the
scheduler (election → allocation → commit): 18 blocking points in 6
categories. Only 3 produce a PEND reason (MATCH_FAIL + 2×NO_RESOURCE);
the limit plugin (4 configured limits), queue state, dispatch_window and
fairshare are configured but silent; 8 LSF-standard reason categories
(dependency/license/reservation/exclusive/worker-state/...) are missing or
stubbed. Includes a layered reason taxonomy and wiring priorities.
Co-Authored-By: Claude <noreply@anthropic.com>
The pending_reason field was declared on models.JobInfo but never populated:
scheduler computed PEND reasons and the controller stored them on
job->pend_reasons, yet the data stopped there. Two gaps closed:
1. controller/apiserver.c: apiserver_job_hook_fun now handles
HOOK_DETAIL_SETPREASON by pushing the updated job to API servers (reusing
apiserver_job_update_fun). Without this, a job whose PEND reasons change
never reached the API server — only JOB_NEW/UPDATE/START/EXIT events did,
and setpreason fired none of those.
2. backend/convert.go + convert_pure.go: cJobToModel / rawJobToModel now
extract xce_job_t.pend_reasons (main_reason + host/res reasons) and
marshal them into models.JobInfo.PendingReason as JSONB. models/jobs.go:
pending_reason added to JOB_DEFAULT_COLUMN so default job queries return
it (it was only in JSONBJFields, never selected).
Verified live on t1/t2/t3: a job with resreq "type == nosuchhost" stays
PEND and returns
"pending_reason":{"main_reason":0,"host_reasons":[{"host":"t1","reason":3},
{"host":"t2","reason":3},{"host":"t3","reason":3}]}
(main_reason=3 PREASON_MATCH_FAIL per host) via both /jobs list, /jobs/:id
and the SSE stream. 4 new unit tests (rawJobToModel pendReason marshal +
name mapping), all apiserver backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
Simplification per review feedback on e3b3ffa: a generic schema-driven
unit field touched the capnp wire schema (2 structs), C structs
(xce_resource_t/xce_resval_t), resobj.c's deep-copy/destroy, opget.c's
reply-build lookup, resources.yaml/resources.schema.json, and mapping.py
-- real cross-cutting cost for a mechanism only one resource (maxmem)
actually needs today.
Reverted all of that. The controller CLI (test.c) now just knows
"maxmem" is bytes-valued via a small explicit is_bytes_valued_resource()
name list, and adaptively picks the best-fit human unit at display time
(format_bytes_human, unchanged) -- same observable behavior (resval/
workerlist render maxmem as e.g. "30.90 GB"), zero wire schema surface.
Regenerated the xceadmin golden fixture back to its pre-unit-field size
(5304 bytes, exactly matching the size right after the mxj-only fix,
confirming a clean, complete removal).
Verified: 1095/1095 controller_unit_test (3 runs; one hit the
pre-existing flaky segfault already logged as bug-1553, unrelated),
228/228 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three bugs found while diagnosing a live controller log, all in the
worker resource-value path:
1. Every host relying on the documented hosts.yaml auto-detect default
(no max_slot, meaning "use CPU count") got 0 scheduling slots,
permanently blocking the scheduler from placing jobs on it. mxj is
UInt32 on the wire and can't carry the schema's -1 sentinel; mapping.py
worked around this by leaving it unset, which defaults to 0 -- and 0 is
never < 0, so apply_join_state()'s auto-detect fallback never fired.
Fixed by encoding the sentinel as 0xFFFFFFFF, which the controller's
`int mxj` field reinterprets back to -1 via its existing implicit
uint32_t->int conversion. No C-side or wire schema change needed for
this part; the fallback itself was already correct but had zero test
coverage for a negative mxj, now added.
2. update_resvals()/update_resval_list() logged the failure reason via
PERROR, which reads process-global errno -- but update_resval()'s
failure paths return raw negative codes directly and never touch
errno, so the controller log filled with misleading "update_resval:
Success" / "update_resval: Resource temporarily unavailable" lines
instead of the real reason. Fixed to decode the actual rc via
xce_strerror_r and name the failing resource.
3. maxmem is canonically stored/wired as raw bytes but had no unit
metadata, so resval/workerlist printed it as a bare number that read
as garbage (e.g. workerlist's fixed-width columns visually smeared a
multi-GB byte count across the MaxCPU/MaxMem headers). Added an
optional `unit` field to the Resource/ResourceVal capnp structs (and
xce_resource_t/xce_resval_t/resources.yaml) so resval and workerlist
can render a human-readable value (e.g. "30.90 GB") while the stored
value stays raw bytes -- display-only, every other resource unaffected.
Regenerated the xceadmin golden fixture (fixtures/full/expected.bin)
twice: once for the mxj sentinel encoding change, once for the new
Resource.unit field growing every resource's wire size by one pointer
slot. Both diffs verified byte-for-byte isolated to exactly the expected
change before regenerating.
Verified: 1095/1095 controller_unit_test (stable across repeated runs),
229/229 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Two related findings logged to .wolf/buglog.json but deliberately not
fixed here (out of scope per discussion): a pre-existing flaky segfault
in the combined controller_unit_test binary, and maxswap/maxtmp being
hardcoded stub values on the worker side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The controller acknowledges EVERY dispatched job through
NotifyData_data_jobDispatchFail (controller/job.c handle_jobdispatch):
a negative code is a real rejection, code 0 is the success ack (the else
branch calls notify_dispatch(jobid, 0)). sched_handle_notify treated any
jobDispatchFail as a failure and returned the job to PEND, so every
successfully dispatched job was immediately re-PENDed, re-elected, and
re-dispatched while the original instance kept running — spawning
duplicate instances cluster-wide. On a 3-worker cluster 24 submitted jobs
ran 38 instances and piled onto the first worker (t1) because its slots
were counted for each duplicate.
Fix: only sched_requeue_pend_job() when fail->code < 0; consume the
success ack (code 0) as a no-op — the job is already RUN
(sched_commit_decision marks it at commit). This was a regression of the
bug-1494 re-PEND logic (bug-1510 fix) that lacked the code guard.
Verified live on t1/t2/t3: 24x sleep-300 jobs distributed exactly 8/8/8,
zero duplicates, all DONE. Add white-box test
test_handle_dispatch_fail_requeues_only_on_negative_code.
Co-Authored-By: Claude <noreply@anthropic.com>
Recovered .wolf/buglog.json (828 entries lost when this worktree's copy
silently reset to bug-001 sometime after commit ce96fd6d, 3 days
uncommitted with zero overlap detection) by merging the committed
history back in and renumbering the 43 real entries logged since.
To stop this recurring:
- OPENWOLF.md: require searching for an existing same-file/root_cause
entry before appending a new bug id (fixes the bug-1418 x4 duplication
pattern found in the recovered history); never restart the id counter.
- stop.js: validate buglog.json every turn (JSON validity, {version,bugs}
wrapper, duplicate ids, count/max-id not regressing vs. the last
commit) and auto-commit it standalone when it grew cleanly, so git
history is always a recoverable backup instead of a 3-day-stale one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Final regression sweep (task 5.4) confirms zero new regressions across
the full stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli)
and proves the single-candidate path is identical to the multi-candidate
path across all three components (Req 1.5), closing out all 21 tasks
(1.1-5.4) of component-failover-adoption.
Two production bugs found during live E2E validation (5.1/5.2) remain
open and out of this spec's scope: bug-039 (apiserver idle-period rejoin
structurally unobservable on a real single-election failover) and
bug-040 (job pushes not reaching apiserver's local store, likely
downstream of bug-039).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live E2E of apiserver's backend-health circuit breaker (task 4.5):
killed all 3 controller nodes (not just the primary, unlike 5.1/5.2 --
this scenario needs a call to genuinely exhaust its retry budget, not
successfully rotate to a survivor). A write call burns through the
20-attempt budget in ~3ms (ECONNREFUSED classifies as zero-backoff
XCE_FO_RETRY_NOW, not the ~70-100s backoff track reserved for a
reachable-but-electing controller), returning the real -E_XCE_NOT_READY
(-10013) and triggering MarkDegraded(10s). Immediately after, a second
write call fast-fails in ~1ms via the literal ErrBackendUnavailable
string -- proving the real C call was skipped entirely, not just fast
-- while a concurrent GET /ping (confirmed by source read to never
touch the backend) keeps succeeding normally throughout.
Cross-corroborated across two independently-generated log subsystems
(apiserver's Go zap log and libxce's own C-level retry log) matching
to the millisecond. Bonus: confirmed the 10s TTL's half-open auto-clear
correctly re-attempts the real call afterward.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Heartbeat-only idle-period E2E (apiserver alone, zero write traffic
during a 268s+ post-kill observation window) designed as a targeted,
cleaner confirmation experiment for task 5.1's bug-039 finding.
Result: bug-039 reproduces a second time, more cleanly than task 5.1
-- zero rejoin log lines throughout the observation window, and even
a post-window diagnostic write call (which itself succeeds against
the confirmed-new primary) triggers zero rejoin activity, ruling out
"task 5.1 was just unlucky timing." Controller-side logs corroborate
independently: zero "apiserver" references anywhere in the new
primary's log, meaning no join attempt of any kind ever reached it.
This task's own literal pass bar was not met -- approved on the same
basis as task 5.1: the validation itself succeeded rigorously and
honestly surfaced a real, now doubly-confirmed gap in the already-
shipped task 4.3/4.4 rejoin mechanism, which is the task's actual
purpose. A dedicated follow-up to fix heartbeat.go's transition
detection is warranted, separate from this spec's remaining scope.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
The MCP/apiserver returned the raw 64-bit job ID (e.g. 73930964699447296)
while xce_test/xcecli display the dash form (e.g. 1-0-1786585485-0) that
parse_jobid accepts. A user taking the numeric form into xce_test fails to
parse it. The apiserver is the data-correctness owner (MCP only reads), so
it now emits the controller's fmt_jobid form alongside the numeric ID:
- models: JobSubmitResponse gains job_id_human; job list rows gain
job_id_human (processJobData), using the existing backend.FormatJobID.
- schedulerclient: URMJob.HumanID + SubmitJobResult.JobIDHuman parse it.
- Tests: backend.FormatJobID (real t1 job, pre-cluster1 job, array child);
processJobData adds job_id_human.
Also rename the SSE permission event:subscribe → event:view so mcp_readonly
stays strictly *:view (the read-only invariant test enforces).
Verified live on t1: scheduler_submit returns job_id_human 1-0-1786586474-0
and scheduler_get_job returns human_id.
Co-Authored-By: Claude <noreply@anthropic.com>
Live 4-process E2E run (3-node controller cluster + xcecd_virtual
worker + xce_scheduler + xce_apiserver, all on the full 3-candidate
list) with the primary killed mid-session. Worker reconnect is
solidly proven with real log evidence and unchanged PIDs across all
three components throughout. Scheduler's rejoin path was never
naturally exercised (no job reached its picker/allocator) -- same
partial-coverage shape already accepted for task 3.3.
Surfaced a likely-real, structural gap in the already-approved task
4.3/4.4 apiserver rejoin mechanism (bug-039): XCECLI_FF_CALL can
silently self-heal the transport within a single heartbeat tick,
making the cross-tick failure->success transition invisible to
heartbeat.go's detector on any failover fast enough to complete
within one heartbeat interval -- which is most real single-election
failovers in this codebase. Also found bug-040 (job pushes never
reach apiserver's store), very likely a downstream symptom of
bug-039's rejoin gap via on_apiserver_join()'s per-process,
unreplicated apiserver registration list. Both logged with detailed,
independently-verified root-cause analysis for a dedicated follow-up
-- fixing them is outside this task's cross-component-validation
boundary.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Audit of apiserver's three task-4.6 coverage areas (degraded-to-healthy
transitions, concurrent-request safety, candidate-list parity):
degraded-to-healthy and candidate-list parity were already fully
satisfied by tasks 4.1/4.5's own tests. Found one genuine gap: reconnect.go's
needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access
test coverage despite being reached from 5 real concurrent HTTP-wrapper
call sites plus the heartbeat goroutine -- health.go's existing
50-goroutine stress test only covered the separate degraded-until
timestamp. One new test closes it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
