bug-1510 (dispatch-fail re-PEND, 6928f74) + bug-1511 (sched_update_workers
in-place update on re-push, 1a2bd3f) together close bug-1494. Update
left_scheduler_work.md and buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
Document that the scheduler dispatch-fail re-PEND fix (commit 6928f74)
resolves one root cause of the "all jobs PEND" symptom; the
controller-restart self-heal-reconnect aspect of bug-1494 remains open.
Log bug-1510 in buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
bug-1506 (controller): sched_worker_change pointed w->host.resvals_to_sched
at resvalist.data, then xlist_fini freed it — a concurrent reader (the
WORKER_JOIN hook encoding the worker's resvals) hit an ASAN
heap-use-after-free that crashed the controller and failed every job. Take
ownership of the resval data via MOVE_PTR (mirrors collect_resvals_workers);
the prior array is intentionally leaked (freeing it races the same readers).
bug-1507 (worker): the apiserver's bug-1500 execuser fix sent the submit
user ("admin") as execuser, but that user does not exist in the worker
host's /etc/passwd. systemd failed the worker unit with EXIT_USER (217),
so every job died immediately after submit. job_start_local_task now
checks the execuser with getpwnam() and falls back to "nobody" when the
user does not exist.
Verified with a full job lifecycle (submit -> RUN as nobody -> DONE) on
the clean build: all 4 daemons healthy, zero ASAN.
Co-Authored-By: Claude <noreply@anthropic.com>
`xcecli jobs list -a` returned "no job found" even when jobs existed.
FilterRows routes every condition on a bitmask field (job_status /
host_status) to MatchesBitmaskCondition, which only implemented OpEQ/OpNE
bitmask semantics — the `-a` filter compiles to `job_status notnull`
(OpNOTNULL), which fell through to `return false` and filtered out every
job. (The default list's `notin(EXIT,DONE)` is compiled to OpEQ(mask), so
it was unaffected.)
Add OpISNULL/OpNOTNULL branches to MatchesBitmaskCondition that test the
raw field value. New query_test.go: 3 regression tests (notnull matches
all, null matches none, bitmask notin path unaffected), RED->GREEN.
Verified live: `xcecli jobs list -a` now lists all jobs including DONE.
Co-Authored-By: Claude <noreply@anthropic.com>
Job PEND root cause when the scheduler registered before the worker:
xcesched_push_workers only fires on scheduler join (sched.c on_scheduler_join),
so a later worker join's slots (set_host_slots from hconfig->mxj=8) is never
re-pushed to the scheduler -> allocation unsatisfied (slots total 0) -> PEND.
Operational fix (no code change): start controller -> worker -> scheduler so
the scheduler's join-time schedworkers push carries the worker's slots.
Co-Authored-By: Claude <noreply@anthropic.com>
The RUSAGE_UPDATE job hook fires with (cluster_t*, xce_job_rusages_t*) as
its varargs — the apiserver and scheduler subscribers parse exactly that
contract — but persist_job_hook_fun's default case treated the second arg
as a job_t* and read j->job.state at offset 56. A xce_job_rusages_t is 56
bytes and shares only jobid@0 with xce_job_t (which is why the jobid read
coincidentally worked), so the state read landed one byte past the struct:
ASAN heap-buffer-overflow (READ of size 4) in thread T7 whenever a job runs
and the worker pushes rusage updates. This was the recurring controller
crash (term 3/4/5) blocking job-PENDING verification.
Add a dedicated HOOK_DETAIL_JOB_RUSAGE_UPDATE case that reads jobid from
the xce_job_rusages_t* and bakes the job's current state from a jobmgr
key_find on the vararg cluster_t* (race-free: the sole caller
update_job_rusages() holds that job's write lock while firing the
AFTER_EVENT hook; re-locking the job inside the hook would deadlock).
Regression test: test_job_hook_rusage_writes_wal_entry now passes a real
56-byte xce_job_rusages_t instead of a full job_t (which masked the
overflow) — ASAN stack-buffer-overflow under the old code, PASS under the
fix. Full controller unit suite: 458 tests, 0 failures.
Co-Authored-By: Claude <noreply@anthropic.com>
modelJobToSubmit built the C job_startinfo_t without setting execuser, so the
controller forwarded a NULL execuser in the startjob request and the worker
ran every job as "nobody". Now the submit carries the resolved exec user: an
explicit exec_user wins, otherwise the job runs as the submitting user.
- add resolveExecUser() (plain-Go, unit-testable) + TestResolveExecUser
- set si.execuser in modelJobToSubmit, free it in freeJobSubmit
Note: deploying this exposed an ASAN heap-use-after-free in the apiserver's
submit/notification path (strncpy reading freed memory); the fix is committed
but needs the memory-lifecycle investigation before it can ship safely.
Co-Authored-By: Claude <noreply@anthropic.com>
put_resvals (resource.c) used `idxs[i] <= 0` to skip "invalid" resource
indexes, but index 0 IS valid — it's the first resource in the table
(e.g. "slots"). For idx 0 the val was never allocated (xvals[i] stayed
NULL) and the write loop then SEGV'd. bug-1495.
Fix: skip only negative indexes (`idxs[i] < 0`), and NULL-guard the
write loop defensively.
Verified end-to-end: a locally-built scheduler now connects to the t1
controller and dispatches all 5 test jobs to node t1 (exec_hosts
populated, jobs run and exit). This was the final blocker for job
scheduling (bug-1494: all jobs PEND).
Also: add the t1 host definition to hosts.yaml (worker join was rejected
with -EINVAL because no matching hostconfig existed for hostname "t1").
Co-Authored-By: Claude <noreply@anthropic.com>
worker/src/resource_sync.c init_builtin_load_indices registers 11 load
indices (r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io) but resources.yaml only
defined r1m/r15s/mem/tmp. The 7 missing ones (r15m/ut/pg/ls/it/swp/io) caused
controller "update_resval: Success" ERRORs and scheduler "unknown resource in
per-worker resvals update" warnings, leaving the worker resource view
incomplete. Add the 7 definitions matching the existing r1m/r15s pattern.
Docs: log bug-1494 (all jobs PEND, scheduler never dispatches; after this
config fix jobs still PEND — remaining suspect is the scheduler→controller
join path after restart), update design doc + verification manual known-issue
tables and root CLAUDE.md cluster-verification status.
Co-Authored-By: Claude <noreply@anthropic.com>
The apiserver returns list rows in OrderMap columnar form —
{"Fields":["col",...],"Data":{"col":val,...}} — but the read clients
unmarshalled each row as a flat object, so every field came back empty
(counts were right, values were all zero/"").
Fix: decode rows as apiRow{Fields, Data map[string]json.RawMessage} and
pull named fields out of Data via rowStr/rowInt/rowInt64/rowFloat helpers.
ListNodes, ListQueues, and getJobs (ListJobs/GetJob) all use apiRow now.
TDD: 3 new ordermap_test.go tests built from live-captured apiserver
response shapes (red: fields empty → green: parsed). Live-verified against
t1: node name=t1 cores=8, 4 queues active, 3 jobs with real fields.
Co-Authored-By: Claude <noreply@anthropic.com>
New §5 in src/docs/chatbot-verification.md: how to connect Claude Code to
the scheduler-agent MCP server. Covers building the mcp-server binary,
`claude mcp add` CLI registration (verified against installed CLI
v2.1.220), the .mcp.json project config schema, the three scopes
(local/user/project), the first-run approval gate, verification commands,
and stdio/env gotchas. Original §5-§7 renumbered to §6-§8.
Co-Authored-By: Claude <noreply@anthropic.com>
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:
1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
library's nextIndex backoff and immediately repl_start_catchup's the
standby at the corrected position, instead of the 16-strike silent
eviction with no catch-up driver after it. Correct from the start, but
live soak still failed identically: nothing ever produced the NACK this
code was waiting for.
2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
silently dropping the reject. ha_send_wal_ack_reply had hardcoded
success=1 unconditionally -- the only WalAckReply sender in the
controller -- making (1)'s NACK-routing correct but unreachable dead
code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
the NACK without a live endpoint. Live-confirmed the NACK now reaches
the primary -- but the scenario still failed for a third reason.
3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
catch-up range was hard-bounded at committed_lsn, which can never advance
in this exact scenario because advancing it requires an ACK from the very
standby that's stuck catching up (2-of-3 alive, this standby is the only
path to a new majority) -- a self-referential deadlock. Widens the
STREAM_WAL ceiling (and the convergence check gating it) to the leader's
own log tail, matching raft.tla's real AppendEntries semantics
(leaderCommit is advisory, never a send-cap), while reusing bug-1478's
exact guard so this doesn't reopen that trap (never widen onto a trailing
local-only checkpoint's own unstreamable LSN).
Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.
Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.
Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.
GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.
Records bug-1461/1462 fixes, admin-config-tool completion, and
config-upgrade-snapshot spec work across anatomy.md, buglog.json,
cerebrum.md, memory.md, and the session/token-ledger state files.
buglog.json: bug-1452 marked FIXED with root cause and fix writeup.
cerebrum.md: Do-Not-Repeat entry for the commit_engine_fini/persist_stop
ordering contract, plus the flag-noted latent duplicate in
test_demotion_convergence.c.
memory.md/anatomy.md/session state: routine session bookkeeping.
Records the controller_unit_test nested UNITY_BEGIN/UNITY_END fix
(commit a81f2a2) in cerebrum.md for future sessions, plus routine
anatomy/memory/token-ledger session bookkeeping.
run-ha-soak.sh: scope every pgrep/pkill to this run's --workdir, not just
binary name + --myname. An unscoped pkill -9 -x xce_controller (old
cleanup()) killed every xce_controller on the machine, including a
different concurrently-running soak's live primary — root cause of the
first bug-1436 misdiagnosis (a killed-out-from-under-it primary produced
a genuine, correctly-reported ECONNRESET that looked like a libxce bug).
.wolf/*: anatomy/cerebrum/buglog updated for bug-1435/1436, session and
token-ledger trail.
buglog.json had been reset to a small auto-detected-noise-only file at some
point this session, losing the real 738-entry history (still present in git
HEAD). Restored the original list verbatim and appended bug-1421 (reusing
the ID a prior session had already reserved for this exact election
regression before the file got wiped) and bug-1422, both logged with full
root-cause/fix detail in this branch's other commits.
memory.md carries the OpenWolf session action log (auto-appended by hooks
plus this session's manual entries) for both fixes.
persist_checkpoint() stamped checkpoint_lsn from p->current_lsn (physical
WAL append tip) instead of p->last_applied_lsn (the true "jobmgr reflects
up through here" cursor), uncoordinated with the async apply pump
introduced by raft-apply-decoupling. write_snapshot() serializes live
jobmgr state, so a checkpoint taken while apply lags the WAL tip mislabels
the snapshot: on restart last_applied_lsn is re-seeded from checkpoint_lsn
and replay resumes strictly above it, silently skipping committed-but-not-
yet-applied entries jobmgr never received. This is the actual mechanism
behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence —
apply_committed_entry() is deterministic/idempotent and write_snapshot()
is crash-atomic, so the bug was a mislabeled watermark, not a structural
divergence requiring a safety/liveness tradeoff.
Fix stamps checkpoint_lsn from last_applied_lsn and holds apply_pump_lock
across the label capture + write_snapshot() so the label and the jobmgr
state it describes are captured atomically w.r.t. the live apply pump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
buglog.json had two broken auto-detection hook signatures polluting it:
- 207 entries "Missing error handling in X" / fix "Added try/catch block"
(nonsensical on .md/.capnp files, and C has no try/catch — never a real
applied fix)
- 452 entries with a literally-empty template ("Significant refactor of "
with no subject ever filled in) / fix "Rewrote N->M lines"
Removed both (659 total). Also merged 24 legitimate duplicate bug IDs
(re-logged/updated entries sharing an ID) into single entries — kept the
most recent content as primary, unioned tags/related_bugs, summed
occurrences — and stripped noise tags ('auto-detected', 'refactor') that
had leaked onto real entries from a since-merged junk duplicate. Net:
1406 -> 721 entries, verified zero remaining duplicate IDs and zero
remaining junk signatures.
.gitignore: added patterns for the scratch soak-run logs and codegraph.html
this session generated at repo root (already deleted from the tree), and
fixed the ucai-formatter-cache.local.json pattern to match its nested
occurrences (.claude/skills/*/.claude/, src/*/.claude/) instead of only the
repo-root one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
