libxce-alloc-preason-equality (requirements, design, tasks; not yet approved):
- alloc_eq compares `from` and excludes only instid; the scheduler stamps
`from` with the deciding session's start, the same clock reading that
is recorded as beginRound, so a replay reproduces it.
- every comparator takes an optional char** why: a malloc'd description
of the first difference, owned and freed by the caller; --sim-verify
mismatch lines print it (resolves bug-1881).
- job_preason_eq is new full-match behavior, not a relocation (bug-1845
keeps the simulator main-reason-only); the three white-box test files
switch to it and keep their sim_verify.c include.
- alloc_eq stays separate from merge_job_alloc()'s same_alloc_slot()
(bug-1863), which does match on instid.
scheduler-sim-discrete-clock (initialized): remove --sim-speed so the
simulator clock moves only at recorded timestamps and a replayed
session_start equals the recorded beginRound exactly.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make_alloc() and make_host_alloc() assigned strdup() results unchecked,
so under memory exhaustion they returned a record whose resname/reserve/
host was NULL; merge_job_alloc()'s same_alloc_slot() (bug-1863) then
strcmp()'d it (ASAN SEGV at xcealloc.c:337). make_alloc() also passed a
NULL resname straight to strdup().
- xcealloc.c: make_alloc()/make_host_alloc() reject a NULL name (EINVAL)
and check every strdup(), freeing what they built and returning NULL
(ENOMEM); on failure make_host_alloc()'s caller keeps `allocs`.
dup_host_alloc() frees its partial copy on failure instead of leaking.
- scheduler.c sched_make_allocs(): a record that cannot be built fails
the whole array (NULL, ENOMEM) instead of leaving a NULL entry.
- seqjob/parjob my_alloc(): check every constructor; on failure free
what was built (parjob also rolls back the remaining claims) and
return -ENOMEM with ctx->alloc NULL -- the caller moves ctx.alloc onto
the job whatever the rc.
Tests: new xcealloc_oom_test (8, --wrap=strdup) and
sched_make_allocs_oom_test (real scheduler_srcs + --wrap=strdup); a new
-ENOMEM test in the seqjob and parjob allocator suites. All RED before
(SEGV / "Expected NULL" / "Expected -12 Was 1"), GREEN after.
xce-build.sh all asan: 115/115 suites OK, Go tests OK.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
follower with no leader known; the liveness poll then retries the
election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
then only holds the watchdog's arm time, and the failover gates took
it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).
bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.
Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stack overflows on user input:
- bug-1702 jobrun -m: host segment strncpy'd into a 256-byte buffer
unbounded (ASAN stack-buffer-overflow); now cmdutil_split_host_tasks()
refuses an over-long host.
- bug-1703 jobsub -n: min strncpy'd into a 64-byte buffer; now
cmdutil_parse_ntasks() parses in place, 1 <= min <= max.
- resupdate's 16-entry vals[] had no bound either; a 17th -n overflowed.
Wrong or silent results:
- bug-1574 resupdate type/model: set_resval() had no string case, left the
entry's name uninitialized, and crashed the encoder. Moved to
resval_typeresolve.c as resval_fill(), which fills every type.
- bug-1705 jobsub -x was overwritten (and leaked) by the default exdata;
cmdutil_default_exdata() only fills an unset one. jobsub's
execuser/exechome/execcwd copies leaked on every call.
- bug-1707 a trailing -n without -v was dropped silently; resupdate also
returned 0 after its RPC failed and "succeeded" with nothing to send.
- bug-1708 jobkill -s SIGKILL became signal 0 and was sent.
- bug-1709 jobinfo exited 0 with no -j, a malformed -j, or no hits.
- bug-1710 hello -c 0 sent nothing and reported success.
- bug-1712 a duplicate -v was reported as "no res name found".
- bug-1777 workerlist showed join-time maxmem/maxswap/maxtmp, not the
effective values (hosts.yaml overrides) the reply already carries in
resvals_to_sched and the scheduler uses.
- bug-1793 join's maxswap/maxtmp defaults were 1 KB / 2 KB in bytes; now
1 GiB / 2 GiB, and -m/-s/-t no longer overflow atoi() above 2 GB.
Error reporting and leaks:
- bug-1796 RPC failures printed perror() of a stale errno ("Connection
refused" / "Invalid argument" / "Inappropriate ioctl for device" for
the same failure). cmdutil_rpc_error() prints the RPC's own code
(E_XCE_* included) at all 19 sites.
- bug-1797 qctrl leaked the queue name on the RPC-error path; repeated
-q/-h leaked the earlier copy.
New src/harness/cmdutil.c (+ cmdutil_unit_test, 14 cases); 9 new
test_test_cli.c cases, 8 RED on the old xce_test; new resval_fill,
worker_effective_attr and join-default tests (join ones RED on the old
join_worker_args.c). Harness suite 6/6 (asan), harness_unit_test 141/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
add_worker strndup's launch_type and the four queue/job pre/post-exec
commands out of the startinfo, but remove_worker only freed exechome,
execcwd, execuser, job_cmd, pids and exdata, so every finished job leaked
up to five strings (LeakSanitizer at xcecd_virtual exit reported the
launch_type one).
New test in test_finished_job_retention.c compares ASAN's live-heap byte
count across an add/remove round trip (the worker suite runs with
detect_leaks=0): RED leaked exactly the 58 bytes of those five strings,
GREEN after. Worker suite 12/12.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
buglog: bug-1433, bug-1450, bug-1451, bug-1469, bug-1474, bug-1728,
bug-1741 fixed; bug-1870 note (load_snapshot off-by-one follow-up,
cb1415d7); new bug-1871 (investigating): snapshot transfer to a rejoining
standby can stall mid-stream after an endpoint unblock.
run-ha-soak.sh --force-snapshot: poll up to 30s for 'HA: snapshot' instead
of one grep right after 3-of-3 convergence, which can be reported before
the last chunks are read (2 of 4 runs failed spuriously; 2 reruns with the
poll passed, installs 4-6 ms after the send).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
schedAdmin, jobStateQuery, jobStateReport and the new notifyAcked
(bug-1741) all logged as "opcode unknown" in the controller's request
trace -- e.g. the bug-1741 factstream soak showed 373 "unknown" requests
that were really acknowledged job reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.
- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
answered. A separate opcode because replies are matched through the
client's single wbuf, not by sequence number: answering plain notifies
would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
do_notify, then waits on the commit barrier and replies 0 only once it
is majority-committed. A barrier failure (the fact is still appended and
retained) replies -E_XCE_NOT_READY, so the worker retries in place
instead of rotating away from a healthy primary (cf. bug-1850). Admitted
by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
(a controller predating the opcode) they fall back to the plain
fire-and-forget notify, so workers and controllers can be upgraded in
either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
retry budget (the rejoin resync remains the backstop). These calls run
on the worker daemon's IPC callback, so while the controller lacks
quorum other jobs' IPC messages wait (buffered, not lost) up to the
retry budget.
Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Regression from 63fd3d2a (bug-1870), found by the --config-upgrade HA soak
(force-elect: "submit on {ctrl-2,ctrl-1} failed", ~640 standby NACKs;
bisected: 725301dd passes, 88c4310e fails).
load_snapshot() seeded current_lsn = checkpoint_lsn, but current_lsn is
the NEXT lsn to write and the snapshot covers 1..checkpoint_lsn. The
off-by-one was always there, hidden because every checkpoint also wrote a
WAL_CHECKPOINT marker ABOVE the checkpoint, so the WAL scan moved
current_lsn past it. With the marker gone, a standby whose WAL held
nothing above its config-marker checkpoint (LSN 11) recovered with
current_lsn=11: its raft layer counted entry 11 as held (ae_already_have,
not written), and persist_wal_write_replicated() then refused entry 12 as
a gap (ae_append_persist_fail) on every resend -- stuck at 11 forever.
Both of load_snapshot's commit points now set current_lsn =
checkpoint_lsn + 1, matching the apply-side catch-up path (lsn + 1).
New test_recover_from_snapshot_without_wal_resumes_after_it (RED before:
current_lsn 3, not 4). Two snapshot-entity tests asserted the old
current_lsn == checkpoint_lsn and now expect +1. Controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_queues() only ran at boot, so a queue a live config upgrade added
had no runtime object in c->qmgr (and the scheduler/apiserver never heard
of it) until the controller restarted. The checkpoint-ordering half of
bug-1469 was fixed in 959b0cda; this is the remaining half.
- reconcile_config_queues(c, notify): creates every queue in the active
config that has no runtime object, leaves existing ones untouched,
returns the number created.
- config_activate_committed() calls it (with notify) after every
successful activation -- verified-marker and no-marker paths alike.
- persist_notify_queue_updated_no_wal(): fires HOOK_DETAIL_QUEUE_UPDATE
under the t_apply_seam_notifying guard, so the scheduler/apiserver are
told while no WAL_QUEUE_CTRL is appended -- config queues are
config-owned, built on every member from the committed config.
- build_queues() now delegates to it without notify, so boot and
activation are idempotent in either order.
New test_config_queue_reconcile.c (RED before: the added queue was never
created): creates the added queue, leaves the existing object untouched,
notifies once, and a second reconcile/build_queues adds nothing.
Controller suite passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Root cause of the "spurious duplicate reconnect" (angle a): zarray_del()
NULLs the slot and decrements count without compacting, so a live entry
can sit at any index < caps. Every standby_endpoints loop bounded by
`count` missed it. Live evidence (--config-upgrade-wal-trim soak,
2026-09-26): ctrl-2 won, promoted vote eps [ctrl-1 (dead), ctrl-3]; the
dead one's removal left [NULL, ctrl-3] with count 1, so the primary
- stopped heartbeating ctrl-3 (primary_heartbeat_fn's send loop),
- re-dialed it ("peer[2] not connected -- initiating outbound connect"
9s later, for a peer that never dropped) and registered a second,
"late-joiner" channel that burned a MAX_STANDBYS repl slot,
- could not evict the stale channel (on_standby_connected's dedup scan),
- and skipped it in config pushes and the ctrlhello broadcast, where a
hole below count also handed endpoint_ref() a NULL.
The re-dial landed in the freed slot 0, which is what made ctrl-3 visible
again -- the "spurious" reconnect was masking the lost heartbeats.
All index scans of standby_endpoints (ha.c heartbeat, sweep, dedup, two
removals; cluster.c removal and ctrlhello; config_repl.c push collection)
now run to `caps` and skip NULL slots; the push collects exactly the
non-NULL entries. `count` stays where it means "how many connected".
New test_reconnect_dedups_channel_behind_deleted_slot (RED before: the
stale channel behind the hole was never evicted). Controller suite 1241/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
