Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
5.6 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1699 | fixed | 2026-09-24 | controller | crash | 2026-08-26T07:35:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1699: ctrl-1 segfaulted (SIGSEGV, core dumped -- confirmed via bash's own accurate job-control message…
Description
ctrl-1 segfaulted (SIGSEGV, core dumped -- confirmed via bash's own accurate job-control message, not a misread) within ~1.5s of a genuinely fresh boot, immediately after processing its first client 'ctrlstatus' probe request, during a fresh 10-round --chaotic run-ha-soak.sh --chaotic invocation (build.dbg, no ASAN) launched to verify bug-1696. Driver reported 'FAIL: no primary elected within ~90s' as a consequence (ctrl-1 never came back; ctrl-2 was subsequently Killed by cleanup).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
NOT ROOT-CAUSED -- occurred exactly ONCE and could not be reproduced despite two dedicated repro attempts: (1) re-running the identical build.dbg chaotic soak under run-ha-soak.sh's built-in GDB_WRAP=ctrl-1 debug seam caught only a benign SIGPIPE (gdb's own default stop-on-SIGPIPE behavior firing on a normal peer-disconnect-during-TLS-write event that the real code already ignores via signal(SIGPIPE, SIG_IGN) at netio.c:1250 -- confirmed this was a gdb artifact, not the real crash, since production code never sees this signal); (2) a manual gdb session with handle SIGPIPE nostop noprint pass set before run (correctly letting SIGPIPE pass through), driving 15+ rapid ctrlstatus probes against a freshly-booted ctrl-1 via xce_test's REPL (mirroring the exact probe pattern run-ha-soak.sh's own wait-primary/discover_primary functions use), did NOT reproduce any crash. (3) A full, from-scratch 10-round --chaotic soak (33 jobs, 10 kill/elect cycles) run under a fresh ASAN rebuild (src/build-asan-1696repro) completed with a clean PASS (committed_divergence=0 committed_job_loss=0) and no SIGSEGV/ASAN report at all. This strongly suggests either a genuinely rare, narrow-window race (possibly specific to cold dynamic-linker/first-request timing that a debug or ASAN build's different memory layout/timing perturbs away) or an environment-specific fluke from that one specific boot -- NOT a deterministic, reliably-triggerable defect. No core file was accessible for postmortem gdb analysis (systemd-coredump is configured as the core_pattern handler but coredumpctl is not installed/reachable in this sandbox). FURTHER NOTE (2026-08-26): did not recur across the same 20+ reproduction attempts made while investigating bug-1698 (fresh boots, standby kill/restart cycles, load, exact user recipe replay) -- still zero occurrences since the single original sighting. FURTHER NOTE (2026-08-26, cont.): USER-SUPPLIED core dumps from the real deployment (4 core.xce_controller.*.zst files today) confirm this is a real, recurring heap corruption (glibc malloc_printerr: "unaligned tcache chunk detected", detected inside endpoint_get()/calloc for a 312-byte endpoint_t -- the abort site is downstream of the actual corruption, not the cause). Confirmed the crash DID occur at least once in this session too (attempt 7 of an earlier 8-attempt loop) but was MISSED by an earlier flawed stderr-grep detection method (setsid nohup ... & disown fully detaches a process from the shells job table, so bash never prints a Aborted/Segmentation fault line for it -- a real methodology gap, not evidence of rarity). Redone with PROPER job-control-based exit-code detection: 15 attempts under a fresh ASAN build (build-asan-1696repro) = 0 reproductions; 15 more attempts with all 3 nodes wrapped in a properly-SIGPIPE-configured gdb session against build.dbg (where it previously reproduced) = 0 reproductions. Total 30 further attempts, zero catches. Hypothesis: ASANs different allocator/timing may suppress the race entirely (consistent with 0/15 under ASAN); build.dbg without a live gdb/gdb overhead may still need many more attempts or a specific concurrent-connection pattern not yet isolated. bug-1698s stuck-reconnect symptom and this heap corruption are plausibly the SAME root cause (both point at endpoint_t lifecycle -- corrupted/UAFd endpoint state could equally explain a connection that never completes ITS handshake). Next step if revisited: run MANY more attempts under build.dbg with proper exit-code detection (not ASAN, which appears to suppress it), or request the user run the real deployment under a debug build with core dumps enabled and share the NEXT core file for backtrace analysis the same way this session did.
Fix
NOT ATTEMPTED -- cannot fix an unreproduced crash responsibly. If this recurs, capture a live gdb session (with SIGPIPE explicitly passed through, per this investigation's corrected GDB_WRAP-equivalent technique) or install/access a working coredumpctl-equivalent BEFORE the crashing process is cleaned up, to get a real backtrace with symbols. Also worth checking core_pattern/ulimit -c in the actual execution environment ahead of time so a future occurrence leaves a usable core file automatically.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1696, bug-1698