Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
10 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | |||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1741 | fixed | 2026-09-27 | 0cad6850: OPCode_notifyAcked -- job start/finish/fail acknowledged after commit, resent until acked; FF fallback for old controllers |
lib | unknown | 2026-08-31T23:45:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
|
true |
bug-1741: User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster + xcecd_virtual worker…
Description
User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster + xcecd_virtual worker, build.dbg): after submitting a 50-element array job (jobsub -a 50 -c hostname) immediately following a fresh primary election, xce_test's joblist shows array elements [6] through [12] (7 consecutive elements) permanently stuck in state RUN (FinishTime=0) while every other element (0-5, 13-49) correctly reached DONE with real FinishTimes, including elements submitted/dispatched both before and after the stuck range and well after the cluster had fully stabilized (up to ~46s post-submit). The user confirmed via repeated joblist checks over several minutes that elements 6-12 never progress -- not a transient lag.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
PARTIALLY ROOT-CAUSED via direct log correlation across worker, controller, and scheduler logs from the live incident (not yet independently live-reproduced in isolation this session -- found while investigating the unrelated bug-1698 HA-hang occurrence in the same incident's data, but happened ~19s BEFORE ctrl-1's hang began and is confirmed unrelated to it: ctrl-1 was fully healthy and primary at the time these 7 finish reports were lost). Timeline reconstructed from three independent logs for elements 6-12: xcecd_worker.log shows the xcecd_virtual master daemon spawned one subprocess per array element (its own established per-job-process architecture), each connecting to the master via a local unix-socket IPC (uv_ipc.c) and reporting 'Completed' for all 7 elements within a ~600ms window (16:25:00.943 to 16:25:01.535). ctrl-1.log shows dispatch_a_job_/on_job_start logged correctly for all 7 elements, but job.c's on_job_finish() (traced via 'job finish jobid ... exit code' TRACE, its own first log line) fired for element [6] ONLY -- elements [7] through [12]'s finish reports never reached on_job_finish at all, i.e. the loss is upstream of job.c entirely, not a hook-firing gap (distinct from bug-1731's fix, which addressed apply_committed_entry() firing HOOK_DETAIL_JOB_START/EXIT -- that fix only helps once on_job_finish/WAL_JOB_FINISH already happened; here it never happens for 6 of 7 jobs). scheduler.log's later repeated 'sched_add_job failed ... rc=-17, merging state into existing tracked job' WARNs for exactly elements 6-12 (recurring at 16:25:23, 16:25:41, 16:25:46 across multiple post-failover JOBS_PUSH resyncs from the new primary ctrl-2) confirm the loss is durable, not just delayed: ctrl-2 inherited ctrl-1's own WAL/replicated state, which never recorded a finish for these 6 elements either, so every resync just re-merges the same stale RUN state -- there is no surviving source of truth anywhere in the cluster that ever learned these jobs finished. Traced the worker-side send path for the likely drop mechanism: xcecd_job_finished() (controller_connector.c:1680) -> xcecli_notify_jobfinish() -> xcecli_notify() (client.c:610, explicitly commented 'Fire-and-forget chokepoint') -> XCECLI_FF_CALL (client_internal.h:290-302), whose own doc comment states plainly: 'single-shot, NOT a retry loop... evaluate EXPR exactly ONCE... A lost send rc is surfaced as-is, never retried.' All 7 near-simultaneous finish notifications for this burst go out over the SAME single xcecli_ctx/network connection (one xcecd master daemon process per worker host, matching this codebase's one-connection-per-daemon model) via this explicitly best-effort, non-retrying path -- exactly the kind of rapid same-connection burst most likely to hit whatever failure mode XCECLI_FF_CALL's own comment already acknowledges as a known risk ('A failed reconnect is non-fatal -- the send below is simply best-effort'). This is the SAME XCECLI_FF_CALL mechanism bug-1548 already flagged for a related-but-distinct reliability gap (silently self-healing the transport across ticks, masking a needed rejoin) -- this occurrence is a second, independent real-world manifestation of the same primitive's fire-and-forget design causing silent data loss, this time for job-finish reports specifically. NOT YET CONFIRMED: the exact internal reason xce_notify()'s single-shot send fails/drops 6 of 7 calls in this specific burst (whether it's a genuine transport error being silently discarded because nothing checks xcecd_job_finished()'s return value, some reentrancy/buffer-reuse issue in xce_notify()'s own encode/send path when called back-to-back without waiting for the prior call's completion, or contention on ac->failover.lock combined with a narrow race) -- would need either a live gdb/strace-instrumented repro of this exact burst pattern, or a close read of xce_notify()'s implementation and whether xcecd_job_finished()'s callers check/log its return code at all (not yet done this session). USER-PROMPTED REFINEMENT (2026-08-31, same session): the user hypothesized this might be a stale-primary artifact (worker kept reporting to the old primary and never resent to the new one). That exact framing doesn't fit the millisecond timing (these 7 finishes/sends happened at 16:25:00.9-01.5, before any failover was detected by anyone), but chasing it surfaced an important adjacent fact from the worker daemon's OWN controller-connection log (xcecd.log, root-owned -- required sudo to read, distinct from xcecd_worker.log's per-job-subprocess-only content used for the rest of this investigation): the WORKER's client-side failover logic logged its own first sign of trouble with ctrl-1 at 16:25:12.164 ('xcecli failover: attempt 1 -> localhost:9901') -- 8 SECONDS BEFORE ctrl-1 itself logged anything wrong ('standby[0] silent for 12s ... evicting', 16:25:20 per bug-1698's own timeline), and about 11-19 seconds AFTER the lost job-finish notifies for elements 6-12 were sent (16:25:00.943-01.535). The worker successfully failed over to the new primary at 16:25:22.332 ('rejoin after transparent reconnect succeeded', following attempt 2 -> localhost:9902 at 16:25:22.165). This reframes the likely relationship between this bug and bug-1698: rather than two unrelated defects that happened to occur in the same incident, the timing is consistent with a SINGLE underlying trigger -- ctrl-1's own I/O/connection handling apparently began degrading measurably before ctrl-1 itself noticed (silently failing to process/ack the worker's job-finish sends circa 16:25:00-01, then dropping the worker's connection entirely by ~16:25:12) -- with ctrl-1's own self-diagnosis (the 12s standby-liveness timeout) simply being the slowest of the three symptoms to surface. If that shared-trigger theory holds, this bug's fix should NOT rely solely on 'the worker eventually fails over correctly' (bug-1698's territory, and confirmed here to work correctly once detected, ~10s after the fact) -- XCECLI_FF_CALL's fire-and-forget send has zero resend/replay mechanism, so a notify lost during exactly this kind of covert pre-failover degradation window is gone forever regardless of how well the subsequent failover behaves. Still not proven whether ctrl-1's degradation genuinely started as early as 16:25:00 (matching the lost notifies) or only at 16:25:12 (matching the worker's own first detected symptom) with the 16:25:00-01 notify loss being a coincidentally-timed SEPARATE glitch -- distinguishing these needs the same live gdb/strace capture bug-1698's own next-steps already call for, ideally instrumented to also watch ctrl-1's handling of inbound OPCode_notify requests specifically, not just its outbound peer reconnect attempts.
Fix
NOT YET FIXED -- found and root-caused-to-this-depth this session while investigating bug-1698 in the same incident's log data; not yet independently reproduced or fixed. Next steps: (1) confirm whether xcecd_job_finished()'s callers (worker_job_callback / process_exit_callback path in xcecd_worker.c) check and log its return code -- if not, that alone is worth fixing regardless of the deeper cause, so a real send failure is at least loud instead of silent; (2) read xce_notify()'s actual implementation (xceapi.c, shared with every other XCECLI_FF_CALL user) to determine whether it is safe to call back-to-back in a tight loop on the same api_ctx_t without waiting for a prior call's underlying I/O to fully complete, or whether it has an internal single-buffer/reentrancy hazard; (3) attempt a targeted live repro (a single worker host completing 5+ jobs within under a second, e.g. a small array of near-instant jobs against one host with concurrency >= the array size) with the worker daemon under gdb/strace to catch xce_notify()'s return code and underlying syscall result for the calls that get lost; (4) once root-caused, likely fix directions include making job-finish notifications durable/retried (not fire-and-forget) given their consequence (a permanently stuck job holding worker slots/resources forever is a correctness issue, not just a display one -- unlike heartbeats, a job-finish report has no self-healing follow-up message the way bug-1737's DONE-vs-EXIT divergence did), or serializing/queuing bursts of notify calls from the same worker daemon so they cannot race or overwrite each other on the shared connection.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1548, bug-1731, bug-1737, bug-1738, bug-1739, bug-1698