Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
41 lines
6.2 KiB
Markdown
41 lines
6.2 KiB
Markdown
---
|
|
id: "bug-1841"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "controller"
|
|
severity: "unknown"
|
|
timestamp: "2026-09-22T09:00:00Z"
|
|
last_seen: "2026-09-22T09:00:00Z"
|
|
occurrences: 1
|
|
files: ["src/controller/resync.c (resync_resolve_round() -- the fix)", "src/controller/persist.c/src/include/persist.h (new persist_notify_worker_status_updated()", "mirroring persist_notify_job_updated_already_persisted()/bug-1679)", "src/controller/testsuite/test_worker_resync.c (new regression test)"]
|
|
tags: ["auto-detected", "resource-leak", "resync", "worker-rejoin-resync", "resInUse", "scheduler", "controller", "hook-notification"]
|
|
related: ["bug-1679", "bug-1799", "bug-1801", "bug-1815", "bug-1812", "bug-1751"]
|
|
legacy: true
|
|
gitea_issue: 75
|
|
gitea_synced_status: "fixed"
|
|
---
|
|
|
|
# bug-1841: User-reported (xce_scheduler -i -s /tmp/scheduler-replay-20260922-071101.replay, worker pc13v, 22…
|
|
|
|
## Description
|
|
User-reported (xce_scheduler -i -s /tmp/scheduler-replay-20260922-071101.replay, worker pc13v, 22 real slots), also reported to occur in normal (non-sim) mode: after worker pc13v rejoined mid-replay (session~162, multiple WORKERS_PUSH+JOB_PUSH bursts, several `JOBS_PUSH: refusing resync state 0x1 for jobid ... existing job is RUN with a real allocation on record` WARNs -- bug-1812's guard correctly refusing a stale PEND resync), every one of that worker's jobs eventually legitimately finished (`info jobs` at session=423 shows all of jobids [0-22] DONE, [23-59] PEND, zero RUN), yet `info worker pc13v` still showed `resource slots: value=22.00 used=22.00` for the rest of the run (session 162 through 423, ~18 simulated minutes) -- the worker was permanently starved even though nothing was actually running on it.
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
The worker-rejoin-resync feature's resync_arm_round() (resync.c) sets XCE_WORKER_RECONCILING on the controller's local worker_t.host.status and (via the HOOK_DETAIL_WORKER_JOIN hook already firing on rejoin) pushes it to the scheduler over WORKERS_PUSH. On the scheduler side, sched_reset_worker_()/sched_update_account() (bug-1799/bug-1801, 2026-09-11) deliberately FREEZE that worker's resInUse -- skip both the per-round zero-reset and the recharge -- while IS_WORKER_RECONCILING is set, by design, so in-flight reconciliation bookkeeping isn't clobbered mid-flight; bug-1799's own fix comment documents that normal reset+resum 'resumes and self-corrects on the very next round' once the status bit clears. But resync_resolve_round() (called from both the report-completion path, resync_apply_report -> resync_maybe_resolve, and the Req-5.3 timeout path, resync_on_timeout) only cleared XCE_WORKER_RECONCILING on the CONTROLLER's own local worker_t -- it never fired HOOK_DETAIL_WORKER_UPDATE or otherwise pushed the corrected status to the scheduler (contrast mark_worker_back()/worker.c:878-888, which does exactly that for the sibling XCE_WORKER_UNAVAIL bit). core_upsert_worker() (scheduler/src/core_events.c) only ever learns a worker's status from whatever WORKERS_PUSH it last received (`worker->status = w->status`, a bare unconditional copy, no other refresh path) -- so the scheduler's copy of the worker's status kept XCE_WORKER_RECONCILING set forever after the controller-side round resolved, permanently freezing resInUse at whatever it was when reconciliation started, even though every job on that worker had long since finished DONE. Reproduces identically in both sim/replay and live/normal mode since both drive the identical xce_core_begin_session() -> sched_reset_worker_()/sched_update_account() freeze logic; sim just replays whatever WORKERS_PUSH events the live controller actually sent, and the live controller never sent the correcting one either. test_worker_resync.c thoroughly tested that RECONCILING clears on the controller's OWN worker_t after resync_resolve_round(), but nothing tested that the scheduler-facing notification actually fires -- a gap noted separately during controller-peers-env-fallback design research (Sep 15 observation: 'full resync path lacks test coverage').
|
|
|
|
## Fix
|
|
FIXED (2026-09-22). Added persist_notify_worker_status_updated(cluster_t* c, worker_t* w) (persist.c, declared in persist.h next to persist_worker_hook_fun) -- the worker-side sibling of persist_notify_job_updated_already_persisted() (bug-1679): fires XCE_CONTROLLER_HOOK_WORKER's HOOK_DETAIL_WORKER_UPDATE AFTER_EVENT under the existing t_apply_seam_notifying guard so persist_worker_hook_fun (also subscribed to this hook, unconditionally WAL-appends on AFTER_EVENT) does NOT durably persist a WAL_WORKER_CTRL fact for this fire -- correct, since XCE_WORKER_RECONCILING/resync_state_t is intentionally ephemeral, re-derived fresh per rejoin, never replayed from WAL. resync_resolve_round() (resync.c) now calls this immediately after clearing XCE_WORKER_RECONCILING and detaching w->resync, guarded so it only fires when a round was genuinely outstanding (never on the already-documented idempotent no-op case). Verified safe under all three call sites' shared precondition (caller already holds w's write lock): confirmed neither HOOK_DETAIL_WORKER_UPDATE subscriber (sched_worker_change/apiserver_worker_change) nor persist_worker_hook_fun (guarded out entirely by the flag before touching w) locks w itself, so no deadlock risk. TDD: added test_resync_resolve_round_notifies_worker_update (test_worker_resync.c) -- a HOOK_DETAIL_WORKER_UPDATE spy hook proves resync_resolve_round() fires it exactly once per genuine resolve and not at all for the idempotent already-resolved no-op. RED confirmed first (Expected 1 Was 0) against the unfixed code, then GREEN after the fix. Full controller_unit_test suite: 1220 Tests, 0 Failures, 0 Ignored, OK (debug profile) -- no regressions. Note: the user's original .replay file is a frozen recording of the pre-fix controller's actual live traffic and will still show the frozen state when replayed, since sim mode only replays already-recorded events; the fix takes effect for new live runs/recordings going forward, not retroactively on old captures.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1679, bug-1799, bug-1801, bug-1815, bug-1812, bug-1751
|
|
- Gitea: #75
|