Add test_sim_replay_reproduces_recorded_alloc_from: a recorded DISPATCH at
an epoch-scale beginRound with a sub-second part replays at the default
simulation settings with no --sim-verify mismatch, and the live allocation's
from equals beginRound in whole seconds. Share the dispatch fixture setup
of the existing tests through two helpers.
Live: 2-round HA soak recordings (11 DISPATCH decisions) replayed with
--sim-verify and no --sim-speed report 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
libxce-alloc-preason-equality (requirements, design, tasks; not yet approved):
- alloc_eq compares `from` and excludes only instid; the scheduler stamps
`from` with the deciding session's start, the same clock reading that
is recorded as beginRound, so a replay reproduces it.
- every comparator takes an optional char** why: a malloc'd description
of the first difference, owned and freed by the caller; --sim-verify
mismatch lines print it (resolves bug-1881).
- job_preason_eq is new full-match behavior, not a relocation (bug-1845
keeps the simulator main-reason-only); the three white-box test files
switch to it and keep their sim_verify.c include.
- alloc_eq stays separate from merge_job_alloc()'s same_alloc_slot()
(bug-1863), which does match on instid.
scheduler-sim-discrete-clock (initialized): remove --sim-speed so the
simulator clock moves only at recorded timestamps and a replayed
session_start equals the recorded beginRound exactly.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make_alloc() and make_host_alloc() assigned strdup() results unchecked,
so under memory exhaustion they returned a record whose resname/reserve/
host was NULL; merge_job_alloc()'s same_alloc_slot() (bug-1863) then
strcmp()'d it (ASAN SEGV at xcealloc.c:337). make_alloc() also passed a
NULL resname straight to strdup().
- xcealloc.c: make_alloc()/make_host_alloc() reject a NULL name (EINVAL)
and check every strdup(), freeing what they built and returning NULL
(ENOMEM); on failure make_host_alloc()'s caller keeps `allocs`.
dup_host_alloc() frees its partial copy on failure instead of leaking.
- scheduler.c sched_make_allocs(): a record that cannot be built fails
the whole array (NULL, ENOMEM) instead of leaving a NULL entry.
- seqjob/parjob my_alloc(): check every constructor; on failure free
what was built (parjob also rolls back the remaining claims) and
return -ENOMEM with ctx->alloc NULL -- the caller moves ctx.alloc onto
the job whatever the rc.
Tests: new xcealloc_oom_test (8, --wrap=strdup) and
sched_make_allocs_oom_test (real scheduler_srcs + --wrap=strdup); a new
-ENOMEM test in the seqjob and parjob allocator suites. All RED before
(SEGV / "Expected NULL" / "Expected -12 Was 1"), GREEN after.
xce-build.sh all asan: 115/115 suites OK, Go tests OK.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
follower with no leader known; the liveness poll then retries the
election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
then only holds the watchdog's arm time, and the failover gates took
it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).
bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.
Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stack overflows on user input:
- bug-1702 jobrun -m: host segment strncpy'd into a 256-byte buffer
unbounded (ASAN stack-buffer-overflow); now cmdutil_split_host_tasks()
refuses an over-long host.
- bug-1703 jobsub -n: min strncpy'd into a 64-byte buffer; now
cmdutil_parse_ntasks() parses in place, 1 <= min <= max.
- resupdate's 16-entry vals[] had no bound either; a 17th -n overflowed.
Wrong or silent results:
- bug-1574 resupdate type/model: set_resval() had no string case, left the
entry's name uninitialized, and crashed the encoder. Moved to
resval_typeresolve.c as resval_fill(), which fills every type.
- bug-1705 jobsub -x was overwritten (and leaked) by the default exdata;
cmdutil_default_exdata() only fills an unset one. jobsub's
execuser/exechome/execcwd copies leaked on every call.
- bug-1707 a trailing -n without -v was dropped silently; resupdate also
returned 0 after its RPC failed and "succeeded" with nothing to send.
- bug-1708 jobkill -s SIGKILL became signal 0 and was sent.
- bug-1709 jobinfo exited 0 with no -j, a malformed -j, or no hits.
- bug-1710 hello -c 0 sent nothing and reported success.
- bug-1712 a duplicate -v was reported as "no res name found".
- bug-1777 workerlist showed join-time maxmem/maxswap/maxtmp, not the
effective values (hosts.yaml overrides) the reply already carries in
resvals_to_sched and the scheduler uses.
- bug-1793 join's maxswap/maxtmp defaults were 1 KB / 2 KB in bytes; now
1 GiB / 2 GiB, and -m/-s/-t no longer overflow atoi() above 2 GB.
Error reporting and leaks:
- bug-1796 RPC failures printed perror() of a stale errno ("Connection
refused" / "Invalid argument" / "Inappropriate ioctl for device" for
the same failure). cmdutil_rpc_error() prints the RPC's own code
(E_XCE_* included) at all 19 sites.
- bug-1797 qctrl leaked the queue name on the RPC-error path; repeated
-q/-h leaked the earlier copy.
New src/harness/cmdutil.c (+ cmdutil_unit_test, 14 cases); 9 new
test_test_cli.c cases, 8 RED on the old xce_test; new resval_fill,
worker_effective_attr and join-default tests (join ones RED on the old
join_worker_args.c). Harness suite 6/6 (asan), harness_unit_test 141/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
add_worker strndup's launch_type and the four queue/job pre/post-exec
commands out of the startinfo, but remove_worker only freed exechome,
execcwd, execuser, job_cmd, pids and exdata, so every finished job leaked
up to five strings (LeakSanitizer at xcecd_virtual exit reported the
launch_type one).
New test in test_finished_job_retention.c compares ASAN's live-heap byte
count across an add/remove round trip (the worker suite runs with
detect_leaks=0): RED leaked exactly the 58 bytes of those five strings,
GREEN after. Worker suite 12/12.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
buglog: bug-1433, bug-1450, bug-1451, bug-1469, bug-1474, bug-1728,
bug-1741 fixed; bug-1870 note (load_snapshot off-by-one follow-up,
cb1415d7); new bug-1871 (investigating): snapshot transfer to a rejoining
standby can stall mid-stream after an endpoint unblock.
run-ha-soak.sh --force-snapshot: poll up to 30s for 'HA: snapshot' instead
of one grep right after 3-of-3 convergence, which can be reported before
the last chunks are read (2 of 4 runs failed spuriously; 2 reruns with the
poll passed, installs 4-6 ms after the send).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
schedAdmin, jobStateQuery, jobStateReport and the new notifyAcked
(bug-1741) all logged as "opcode unknown" in the controller's request
trace -- e.g. the bug-1741 factstream soak showed 373 "unknown" requests
that were really acknowledged job reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
