The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
buglog: bug-1433, bug-1450, bug-1451, bug-1469, bug-1474, bug-1728,
bug-1741 fixed; bug-1870 note (load_snapshot off-by-one follow-up,
cb1415d7); new bug-1871 (investigating): snapshot transfer to a rejoining
standby can stall mid-stream after an endpoint unblock.
run-ha-soak.sh --force-snapshot: poll up to 30s for 'HA: snapshot' instead
of one grep right after 3-of-3 convergence, which can be reported before
the last chunks are read (2 of 4 runs failed spuriously; 2 reruns with the
poll passed, installs 4-6 ms after the send).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gitea-sync.py: --ids syncs named bugs regardless of dates (OR'd with
--since). Bulk triage bumped last_seen/status_updated across most of the
buglog, so --since 2026-09-01 matched ~1150 bugs while recently fixed older
bugs (bug-1798..1804, 1714, 1754, 1850, 1854) could never be reached.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The 16 commits since 534895c5 were re-signed (rebase --exec amend -S; tree
unchanged), which changed their hashes. Update the buglog and the
scheduler-allocation completion note to the new ones.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since bug-1790's fix (9caf1424) counts a job's full task total against
global limits, the repo's limit1 (all users except rshen share 3 slots)
pends xce_test's root-submitted 33-task job forever, so the multihost
e2e fixture failed even though the allocator is correct.
deploy-ha.sh gains an optional XCE_DEPLOY_CONFIG_DIR (default unchanged:
src/configs). run-multihost-verify.sh copies src/configs into its deploy
dir, replaces limits.yaml with an empty list, and deploys with that
override.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- buglog: gitea_issue / gitea_synced_status for the 20 bugs created and the
30 already-synced entries (#25/#35 reopened as partial)
- .kiro/specs/*/spec.json: "gitea" milestone/issue mapping for all 66 specs
(65 milestones + 195 issues created, 158 closed at creation)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog had no status field, so "is this bug open?" meant reading free
text, and gitea-sync closed any issue whose `fix` text was non-empty --
including "NOT FIXED"/"PENDING"/"PARTIALLY FIXED" ones (bug-1855).
- Backfill `status` (+ status_updated) on all 1,193 entries. The 85 entries
whose text said not-fixed/pending/out-of-scope were checked against HEAD
and git history; the rest are `fixed`. Result: 36 open, 10 partial,
2 investigating, 5 cannot-reproduce, 4 wontfix, 1 duplicate.
Insert-only diff: existing lines are untouched.
- .claude/rules/buglog-status.md defines the status values (kept out of
openwolf.md, which `openwolf update` overwrites).
- scripts/buglog.py + buglog skill: add (next free id), status, note,
recur, list, show, triage. Edits only the touched entry's lines; a full
json.dumps rewrite churns ~1,200 lines of mixed \u-escaping.
- gitea-sync (script + skill): state and a single bug:<status> label come
from `status`; reopen supported; issues shared by several entries follow
the most-open one; only status labels are swapped; per-entry write-back
with gitea_synced_status so unchanged entries cost no API calls; list-typed
`fix` no longer crashes; spec issues close on done phases (spec.json has
no implementation approval); --since no longer floods Gitea after the
backfill.
- Disable OpenWolf buglog auto-detection: its hook numbers new entries
len(bugs)+1, which collides with existing ids (bug-1856, upstream).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A chaotic round's walcheck compared all three members' on-disk WALs with
boundary=auto. The just-killed term-4 primary still held its NACKed,
never-committed lsn=58, which differed from the new term-5 leader's
own-term noop at lsn=58, and was counted as class=committed.
walcheck_count now takes an optional "midloop" mode, used only by the
per-round chaotic check: the boundary is the serving primary's own
committedLsn (live_committed_boundary() needs 3-of-3 convergence, which a
chaotic round never reaches), and only members guaranteed to hold the
committed prefix are compared -- the primary plus live standbys whose
appliedLsn has reached the boundary. Every other caller, including the
settled final sweep, still compares all three. Log lines name members=.
Also logs bug-1854 (scheduler re-adds every configured resource on each
rejoin config push; ERROR noise, changed definitions ignored) for
follow-up.
10-round chaotic soak: PASS (divergence 0, job loss 0, SIMVERIFY 9 files,
0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The recorder encoded each controller event batch after
xce_core_push_events() had applied it, and applying moves payload fields
out of the events: a WORKERS_PUSH's resvals (core_apply_worker_resvals
MOVE_PTR), a JOBS_PUSH's allocs, a JOB_RUSAGE's rusages, and a
CONFIG_PUSH's shared resvals (sched_apply_pending_config recorded after
sched_init_config). Recordings silently lost exactly those fields -- a
worker's resval names were written with NULL values, so --sim-verify
replayed "no slots" and turned 3 live DISPATCHes into PENDs.
driver.c now records the batch before the push; the periodic in-stream
SnapshotEntry moves out of sched_record_write_events() into new
sched_record_checkpoint_after_apply(), called after the push so a
checkpoint still reflects the events written ahead of it. The config
push is likewise recorded before sched_init_config().
Regression test: test_recorded_workers_push_keeps_resval_values (RED:
recorded value NULL; GREEN after). Periodic-checkpoint tests updated to
the write-then-checkpoint contract. meson test 109/109; 25 live soak
recordings replay with 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_val_res() memcpy'd the xce_val_t into w->host.resvals, so the
"type"/"model" string resources aliased the interned xstrings owned by
w->host.attrs. rejoin_worker() drops the old attrs strings, then
apply_join_state() -> resync_arm_round() resolves the previous join's
still-outstanding round, firing the worker hooks before
add_hostattrs_resvals() re-seeds resvals; the apiserver hook's
collect_resval() strdup()ed the freed hostmodel. Live: a newly promoted
primary died ~8s after election during an HA soak.
A string resval now stores a private strdup (taken before the old copy is
freed), and the owning arrays free their entries: host.resvals gets
kill_resval (replacing the FIXME NULL killer) and c->sharedinst's
resval_obj_killer (previously a no-op) frees the string and the value --
both also leaked every xce_val_t before. Also closes the same alias for
worker-reported string resvals and config values.
Regression test: test_rejoin_worker_string_resvals_survive_outstanding_round
(RED reproduced the live ASAN stack; GREEN after). meson test 109/109.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follow-ups from the bug-1846 investigation.
bug-1847: config pushes never reached a .replay recording -- handler.c
applied them synchronously, bypassing the pending-update queue the
recorder writes from. With bug-1846's deferral they're now applied at a
round boundary on the driver thread, so sched_apply_pending_config()
writes a CONFIG_PUSH event via the existing sched_record_write_events()
(no-op when not recording), ahead of the round's other recorded events.
Test: test_config_push_deferred.c's test_applied_config_push_is_recorded
(RED: 0 events; GREEN: 1). Fidelity limit, pre-existing: replay applies
configPush through core_events.c's CONFIG_PUSH path, which doesn't re-run
groupmgr_load_configs()/plugin_config_update(), so a limits/groups change
still won't take effect in sim.
bug-1848: xce_sched_obj_t.hlimits was created with a NULL zarray killer
at all three constructors, so reset_sched_obj() and schedobj teardown
leaked the per-worker xlist_t match_job_host_limit_() callocs. New
kill_hlimits_list() (schedobj.c) frees the list, never its vres entries
(owned by the limit plugin); core_events.c and snapshot_load.c use
identical static local copies, since both avoid sharing helpers with
schedobj.c and several test binaries link them without it. Test:
test_limit_host_dedup.c's test_hlimits_zarray_releases_its_lists (RED:
killer NULL; GREEN).
bug-1849: every --sim run warned "failed to resolve job picker plugin ...
picker rebuild will be skipped". Benign: the snapshot is loaded before
the sim registers its picker, and sim_bootstrap_picker() replays the same
queue/job walk afterward. Downgraded to TRACE with an explanation.
Buglog also records bug-1850 (open): the ~100s reconnect churn behind
bug-1846 part 2 -- xcecli_retry_next() rotates/reconnects on every retry
(ETIMEDOUT after the 10s reply deadline, E_XCE_NOT_READY, redirects), so
a slow reply from a healthy primary forces a rejoin + full re-push.
Needs the live scheduler.log and a libxce spec.
Full scheduler suite 87/87 (asan). Replay of the captured file: same 184
mismatches, all attributable to the recording's pre-fix binary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
