lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(buglog): one Markdown record per bug, checked against a record standard

Browse source 
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).

- .claude/skills/buglog/bug-record.md: the record standard. Eight
  sections (Summary as the title, Description, Environment, Reproduction;
  Root cause, Fix, Verification, References once fixed), each with the
  content it must contain -- e.g. Environment needs the OS, the libc the
  failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
  shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
  verified to round-trip. They carry legacy: true and an explicit "not
  recorded" where the old log had no such field; component derived from
  file paths, severity from keywords (else unknown), fix_commits from
  hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
  the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
  the machine and --binary), check, status (refuses a status the record
  doesn't support; fixed needs Root cause, Fix, Verification and
  --commit hashes git resolves), note, recur, set, show, list, triage,
  env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
  records, a legacy record hand-changed to fixed without the fix-time
  sections, a new record claiming legacy: true, or a stale index refuse
  the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
  sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
  override), and a body hash so an issue's text is refreshed when the
  record changes.
- bug-1866 rewritten to the full standard as the worked example.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-28 03:40:11 +00:00
parent 27b7a84980
commit d26ff8870d
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
512 changed files with 40724 additions and 17843 deletions
Download patch file Download diff file Expand all files Collapse all files

42
.claude/rules/buglog-status.md
View file

@ -1,34 +1,34 @@
# Buglog status field
# Bug records and their status
Kept separate from `openwolf.md` because `openwolf update` overwrites that file.
OpenWolf itself ignores this field (search, pre-write hints and the hook-created
entries don't read or set it), so the convention is enforced only by this rule.
- Every `.wolf/buglog.json` entry carries a `status`, and it is the only source
of truth for whether a bug is open. Never infer it from `fix`: many entries
have a populated `fix` that says "NOT FIXED", "PENDING" or "PARTIAL".
- Each bug is a Markdown record, `.wolf/bugs/<id>.md`, written to the standard
in `.claude/skills/buglog/bug-record.md` (what Description, Environment,
Reproduction, Root cause, Fix and Verification must contain). Create and
change records with the `buglog` skill (`scripts/buglog.py`), which checks
them against that standard.
- `.wolf/buglog.json` is an index generated from the records; never edit it.
`.wolf/buglog-archive.json` holds retired OpenWolf hook auto-detections.
- The front-matter `status` is the only source of truth for whether a bug is
open. Never infer it from the Fix text or tags: migrated records often
quote "NOT FIXED" or "PENDING" from earlier stages.
- Values:
- `open`: defect confirmed and still in the code. Intermittent test
failures that do still recur count as `open`, not `cannot-reproduce`.
- `partial`: part fixed; `status_note` says what remains.
- `investigating`: real failure observed, root cause not found yet or
waiting for a reproduction.
- `fixed`: fix committed; cite the commit in `fix` or `status_note`.
- `fixed`: fix committed; `fix_commits` lists the commit(s) and the
Verification section says how the fix was proven.
- `cannot-reproduce`: reproduction was attempted and failed; record what was
tried in `repro_attempts` (e.g. "0/40 runs, 2026-08-01").
- `wontfix`: not a bug, accepted limitation, or deferred with no plan.
- `duplicate`: same defect as another entry; set `duplicate_of`.
- Whenever you log a bug or change its state, set `status` and
`status_updated` (YYYY-MM-DD), plus `status_note` unless the status is
`fixed` with the commit already in `fix`. Put them right after `"id"`.
- A `cannot-reproduce` bug that recurs: bump `occurrences`/`last_seen` and set
`status` back to `investigating`.
- Use the `buglog` skill (`scripts/buglog.py`) to add entries and change
status: `add`, `status`, `note`, `recur`, `list`, `triage`. It picks the
next free id and edits only the touched entry's lines. Don't re-serialize
the whole file with `json.dumps`/`jq`; that rewrites ~1,200 unrelated lines.
- Open-bug list:
- `duplicate`: same defect as another record; set `duplicate_of`.
- Change status only with `python3 scripts/buglog.py status <id> <status>`
(`--note` for every status except `fixed`; `--commit <sha>` for `fixed`).
It sets `status_updated` and refuses a change the record doesn't support.
- A `cannot-reproduce` bug that recurs: `buglog.py recur <id>` bumps
`occurrences`/`last_seen` and sets `status` back to `investigating`.
- Open-bug list: `python3 scripts/buglog.py list --open`, or from the index:
`jq -r '.bugs[] | select(.status | IN("open","partial","investigating")) | "\(.id) [\(.status)] \(.status_note // .error_message[0:90])"' .wolf/buglog.json`
- Entries with no `status` (hand-written without it, or created by an OpenWolf
hook) need triage:
`jq -r '.bugs[] | select(.status == null) | .id' .wolf/buglog.json`
- Records with no valid status need triage: `python3 scripts/buglog.py triage`.

72
.claude/skills/buglog/SKILL.md
View file

@ -1,7 +1,7 @@
---
name: buglog
description: Log a new bug, change a bug's status (fixed / partial / cannot-reproduce / ...), record a recurrence, or list open bugs in the OpenWolf buglog (.wolf/buglog.json). Use whenever you log or fix a bug, or are asked for the open-bug list.
argument-hint: add | status <id> <status> | note <id> | recur <id> | list [--open] | show <id> | triage
description: Log a new bug, change a bug's status (fixed / partial / cannot-reproduce / ...), record a recurrence, or list open bugs. Bugs are Markdown records in .wolf/bugs/<id>.md checked against a record standard. Use whenever you log or fix a bug, or are asked for the open-bug list.
argument-hint: new | status <id> <status> | note <id> | recur <id> | list [--open] | show <id> | check
triggers:
- /buglog
- log a bug
@ -11,12 +11,23 @@ triggers:
# Buglog Skill
Every operation goes through `scripts/buglog.py` (run from the repo root). It
edits only the lines of the entry it touches, re-parses the file before
saving, and writes atomically. **Never** rewrite `.wolf/buglog.json` with
`json.dumps`/`jq`/the Write tool: entries mix raw UTF-8 and `\uXXXX` escapes,
so a re-serialize rewrites ~1,200 unrelated lines. Never read the file whole
either (~27K lines); use `list`/`show`.
Each bug is one Markdown file, `.wolf/bugs/<id>.md`: short machine fields in
front matter, `# <id>: <summary>`, then the sections of the record standard.
**Read `bug-record.md` (next to this file) before writing a record**: it says
what every section must contain (e.g. Environment needs the OS, the libc the
failing binary loads, toolchain, build profile, commit and deployment).
- `.wolf/buglog.json` is an **index generated** from the records (for
OpenWolf's hooks, `openwolf bug search` and `jq`). Never edit it; every
`buglog.py` write regenerates it, and `python3 scripts/buglog.py index` does
it after a hand edit of a record.
- `.wolf/buglog-archive.json` holds the retired OpenWolf hook auto-detections
(710 entries, template messages like "Missing guard clause"). They are
not bug reports; leave them there.
- Records migrated from the old JSON log carry `legacy: true` and
`_Not recorded (migrated from buglog.json)._` where the old log had no such
field. They are exempt from the reporting sections, but changing one to
`fixed`/`partial` still needs the fix-time sections.
Status values and when to use each: `.claude/rules/buglog-status.md`.
@ -29,36 +40,35 @@ or `python3 scripts/buglog.py list --grep '<regex>'`.
| Task | Command |
|------|---------|
| Log a new bug | `python3 scripts/buglog.py add --error "<symptom, exact error text>" --root-cause "<cause>" --file <path>[,<path>] --tags scheduler,asan --status open --note "<one line>"` → prints the new id |
| Log a bug you already fixed | same, with `--fix "<what changed>, commit <sha>" --status fixed` |
| Mark fixed | `python3 scripts/buglog.py status <id> fixed --note "<sha>"` |
| Mark partly fixed | `python3 scripts/buglog.py status <id> partial --note "<what remains>"` |
| Log a new bug | `python3 scripts/buglog.py new --summary "<symptom in component, ≤100 chars>" --component <c> --severity <s> [--binary <failing binary>] [--files a,b] [--tags x,y] [--related bug-N]` → prints the file path. Environment is filled in from this machine (pass `--binary` so libc and build profile come from that binary). Then edit the file: replace every `TODO` in Description, Environment (`Deployment`) and Reproduction, and run `check <id>`. |
| Environment block only | `python3 scripts/buglog.py env [<binary>]` |
| Check records | `python3 scripts/buglog.py check <id> …` / `check --all` |
| Mark fixed | Fill in Root cause, Fix and Verification (`- Test:` RED before/GREEN after, `- Suites:`, `- Live:`), then `python3 scripts/buglog.py status <id> fixed --commit <sha> [--commit <sha2>]`. Refused, and nothing written, until the record meets the standard; each `--commit` must be a real commit. |
| Mark partly fixed | `python3 scripts/buglog.py status <id> partial --note "<what remains>"` (Root cause + Fix required) |
| Couldn't reproduce | `python3 scripts/buglog.py status <id> cannot-reproduce --repro "0/40 runs, 2026-09-24" --note "<what was tried>"` |
| Real failure, cause unknown | `python3 scripts/buglog.py status <id> investigating --note "<next step>"` |
| Not a bug / accepted | `python3 scripts/buglog.py status <id> wontfix --note "<why / who decided>"` |
| Duplicate | `python3 scripts/buglog.py status <id> duplicate --duplicate-of <other-id> --note "<same defect as ...>"` |
| Add findings to an entry | `python3 scripts/buglog.py note <id> "<text>"` (appends an `UPDATE (date):` paragraph to `fix`) |
| Add findings | `python3 scripts/buglog.py note <id> "<text>" [--section "Root cause"]` (appends `UPDATE (date): …`; default section Description) |
| Seen again | `python3 scripts/buglog.py recur <id> [--note "<where>"]` (bumps occurrences/last_seen; reopens a `fixed`/`wontfix`/`cannot-reproduce` bug) |
| Open-bug list | `python3 scripts/buglog.py list --open` (add `--tag scheduler`, `--grep`, `--since YYYY-MM-DD`) |
| One entry | `python3 scripts/buglog.py show <id>` |
| Entries missing a status | `python3 scripts/buglog.py triage` |
| Any other single-line field | `python3 scripts/buglog.py set <id> key=<json> ...` |
| One record | `python3 scripts/buglog.py show <id>` |
| Front-matter field | `python3 scripts/buglog.py set <id> key=<json> ...` |
`status` requires `--note` for every status except `fixed`, and always
rewrites the note, so a stale note never outlives its status.
Components: controller, scheduler, worker, lib, harness, apiserver, xcecli,
xceadmin, autoscaler, build, config, tooling. Severities: data-loss, crash,
hang, wrong-result, leak, security, noise.
## Rules
- **Ids**: let `add` pick the id (one past the highest existing). Don't number
entries by hand or by entry count. OpenWolf's post-write hook numbers by
entry count, which collides with existing ids; its auto-detection is turned
off in `.wolf/config.json` for that reason (`openwolf.buglog.auto_detect`).
- **Write the symptom first**: `--error` holds the observed error text or
behavior, not a status banner ("OPEN, NOT FIXED --" goes in `status`).
- **Fixing a bug in code**: after the fix commit, run `status <id> fixed
--note <sha>`. If only part is fixed, use `partial` with what remains.
- **Arrays** (`tags`, `related_bugs`, `files`) can only be set at `add` time;
to change them later, edit that entry's lines by hand and re-check with
`python3 -c "import json; json.load(open('.wolf/buglog.json'))"`.
- **Gitea**: status changes reach Gitea on the next `/gitea-sync`, which opens,
closes or reopens issues from `status`.
- **Ids**: let `new` pick the id (one past the highest in records + archive).
- **Editing sections**: edit the Markdown file directly (Edit tool), keeping
the `## <Section>` headings; then `check <id>` and `index`. Status changes
go through `status`, never a hand edit of `status:`.
- **Enforcement**: the pre-commit hook (`scripts/git-hooks/pre-commit`; install
once per clone with `ln -sf ../../scripts/git-hooks/pre-commit
.git/hooks/pre-commit`) runs `check --staged`: it refuses staged records
that miss the standard, a legacy record hand-changed to fixed/partial without
the fix-time sections, and a stale `buglog.json`.
- **Gitea**: `/gitea-sync` mirrors status, and refreshes an issue's title and
body whenever the record's text changes; fix commits are linked.

318
.claude/skills/buglog/bug-record.md Normal file
View file

@ -0,0 +1,318 @@
# Bug Record Standard
Each bug is one file, `.wolf/bugs/<id>.md`. `scripts/buglog.py` creates
records, changes their status and checks them against this standard; the
pre-commit hook refuses a commit whose staged records do not meet it. See
[Storage and enforcement](#storage-and-enforcement).
Every bug record is read by two audiences: a person deciding what to work on,
and an agent that has to reproduce, fix or verify the bug with no other
context. So a record uses **few fields**, and each field is plain Markdown
text that must satisfy the rules in its section below. The rules say what the
text must contain, how to collect it, and what does not count.
Write every field so that someone who has never seen the bug, on a different
day and a different machine, could act on it.
## Fields at a glance
| # | Field | When required | One-line rule |
|---|-------|---------------|---------------|
| 1 | **Summary** | always | The `# <id>: <summary>` title: one line, ≤ 100 characters, symptom + component. |
| 2 | **Description** | always | What happened vs. what should happen, with the exact error text and the impact. |
| 3 | **Environment** | always | OS, libc, toolchain, build profile, deployment, and the exact code version. |
| 4 | **Reproduction** | always | Steps someone else can run, how often it fails, and where the evidence is kept. |
| 5 | **Root cause** | status `fixed` or `partial` | The mechanism, in code terms, that produces the symptom. |
| 6 | **Fix** | status `fixed` or `partial` | What was changed and why this change removes the cause. |
| 7 | **Verification** | status `fixed` | Proof the fix works, and that the same test failed before the fix. |
| 8 | **References** | status `fixed` | Fix commits, related bugs, specs or docs. |
Machine fields stay short and live in the front matter, outside these
sections: `id`, `status`, `status_updated`, `status_note`, `component`,
`severity`, `timestamp`, `fix_commits`, and a few more. Their values are defined in `.claude/rules/buglog-status.md`
and in [Machine fields](#machine-fields).
---
## 1. Summary
**Must include**
- The observable symptom, not the suspected cause.
- The component or binary where it shows (`xce_controller`, `xcecd_virtual`,
`xce_scheduler --sim`, `gitea-sync`, …).
**Rules**
- Written as the record's title, `# <id>: <summary>`; the summary itself is
one line, ≤ 100 characters, no trailing period. It becomes the Gitea issue title.
- No status words (`OPEN`, `FIXED`, `NOT FIXED`, `PRIORITY: LOW`). Status lives in `status`.
**Not acceptable**: `Missing guard clause` · `Incorrect value in code` ·
`crash` · `bug in ha.c`.
**Good**: `xcecd_virtual leaks worker strings on every finished job (LeakSanitizer at exit)`
## 2. Description
**Must include**
- **Observed**: what happened, with the exact error text, log line, assertion
or sanitizer headline in a code block, copied, not paraphrased.
- **Expected**: what should have happened instead.
- **Impact**: who or what is affected and how badly: crash, hang, data loss,
wrong result, leak (and its growth rate), noise only.
- **How it was found**: user report, HA soak, ASAN run, audit, code review,
unit test, or a spec task (name it).
**Include when relevant**
- The first bad state you can see, when the visible symptom comes much later.
- Which other bugs showed up in the same run.
**Rules**
- Keep it to the facts known at the time you log the bug. Later findings go
in Root cause, or in a dated `UPDATE (YYYY-MM-DD):` line at the end of this
section. Never rewrite history silently.
- No guesses stated as facts. Label a suspicion as `Suspected:`.
## 3. Environment
The goal: another person can tell whether their machine and build are
affected, and can recreate the same conditions.
**Must include**
- **OS**: distribution and version, and the kernel.
- Kernel: `uname -srm`.
- Distribution: `/etc/os-release` `PRETTY_NAME`. If that file is missing,
say so and give what identifies the system instead. For example, this dev
host has no `/etc/os-release`. It is a Docker container (`/.dockerenv`
exists) on a NixOS kernel (`/proc/version` shows `#1-NixOS`).
- Container or VM, when it applies.
- **libc**: name and version of the C library **the failing binary actually
loads**, not the host's.
- Get it from the binary:
`readelf -l <binary> | grep 'program interpreter'` → `/nix/store/…-glibc-2.42-47/lib/ld-linux-x86-64.so.2`.
- Why it matters: on this host `ldd --version` outside the Nix shell reports
glibc **2.40**, but every XCE binary built in the dev shell runs on glibc
**2.42**. The host value is the wrong answer.
- **Toolchain**: compiler and version (`gcc --version`) plus Meson, and Go or
Python for those components, taken from the same shell that built the binary.
- **Build**: profile and directory: `asan` (`xce.build`), `debug`
(`build.dbg`) or `release` (`build.release`); and the sanitizer options in
force when the failure was seen (e.g. `ASAN_OPTIONS=detect_leaks=0`).
- **Code version**: the commit the failing binary was built from, and the
branch (`git rev-parse --short HEAD` at build time; add `-dirty` if the tree
had uncommitted changes). For a bug seen on a deployed cluster, the version
of **each** component involved (controller, scheduler, worker, apiserver,
xceadmin) when they differ.
- **Deployment**: topology (single controller, 3-node HA, multihost), worker
kind (`xcecd_virtual` or real root `xcecd`), and host names when more than
one machine is involved.
**Include when relevant**
- Config set (`src/configs` or a fixture directory) and any non-default values.
- CPU count and memory, for races, timeouts and load-dependent bugs.
- systemd and cgroup version, for worker process or accounting bugs.
- Anything else that ran on the machine at the same time (parallel builds,
other soaks), for timing-sensitive failures.
**Not acceptable**: `Linux` · `local` · `latest code` · `ASAN build` with no
commit · the host's `ldd --version` for a Nix-built binary.
**Collect it with** (Nix-built binaries):
```bash
uname -srm; cat /etc/os-release 2>/dev/null | grep PRETTY_NAME || echo "no /etc/os-release"
[ -f /.dockerenv ] && echo "docker container"
readelf -l <binary> | grep -o '/nix/store/[^]]*ld-linux[^]]*' # libc actually loaded
nix develop ./src --command bash -c 'gcc --version | head -1; meson --version'
git rev-parse --short HEAD; git diff --quiet || echo dirty
```
## 4. Reproduction
**Must include**
- **Steps**: numbered commands someone else can paste, starting from a clean
checkout at the commit in Environment. Name the test binary, soak mode,
replay file or script and its arguments.
- **Frequency**: `deterministic`, or measured as `N/M runs` with the date
(e.g. `2/4 force-snapshot runs, 2026-09-26`). Never `sometimes`.
- **Evidence**: where the logs, cores, replay files or captures are kept. Use
a path that survives the session: copy them out of `/tmp` and out of
per-session scratch directories first. If they could not be kept, say so.
**Include when relevant**
- A smaller reproduction (unit test, single command) once you have one. Keep
the original steps too.
- Conditions that stop it reproducing (e.g. passes with `--jobs 1`).
**When it does not reproduce**: record what was tried, how many runs, on
which commit and date; this is what `status cannot-reproduce --repro` stores.
## 5. Root cause *(fixed or partial)*
**Must include**
- The mechanism in code terms: the function(s) and file(s), the state or
ordering that goes wrong, and why that produces the symptom in Description.
- Since when: the commit or change that introduced it when known
(`git log -S`, `git bisect`), otherwise `pre-existing, origin unknown`.
- Scope: other call sites or components with the same defect, checked or
not checked.
**Rules**
- Describe the cause, not the fix. `Added a NULL check` is a fix, not a cause.
- If part of the explanation is still unproven, mark that part `Unconfirmed:`.
## 6. Fix *(fixed or partial)*
**Must include**
- What changed, per file or function, in a few lines.
- Why this change removes the cause in Root cause.
- For `partial`: what is left and where it is tracked.
**Include when relevant**
- Alternatives that were rejected, and why.
- Behaviour changes, compatibility or migration notes (wire format, WAL,
snapshot, config).
**Rules**
- A proposal that is not implemented yet is labelled `Proposed:` and needs
status `open` or `investigating`, never `fixed`.
## 7. Verification *(fixed)*
**Must include**
- **Test added or changed**: the test name and file, and a statement that it
**failed before the fix** (RED) and passes after (GREEN). If no automated
test is possible, say why and describe the manual check.
- **Suites run**: which suites, on which build profile, with results
(e.g. `worker suite 12/12, asan`).
- **Live check** for HA, persistence, scheduler or worker-process bugs: the
soak or deploy run used, with its mode, rounds and result.
**Not acceptable**: `tests pass` · `verified` · `should be fixed now`.
## 8. References *(fixed)*
**Must include**
- Fix commit(s), short SHA, on a pushed branch. They live in the front-matter
field `fix_commits`, set by `buglog.py status <id> fixed --commit <sha>`,
which refuses a hash that is not a commit in this repository.
**Include when relevant**
- Related bugs, with the relationship: `regression of`, `split from`,
`split into`, `same root cause as`, `duplicate of`.
- Spec, steering or design documents that changed or explain the design.
- The Gitea issue number (filled in by `gitea-sync`).
---
## Machine fields
Front matter: one `key: <JSON value>` per line (valid YAML), between `---` lines.
| Field | Values |
|-------|--------|
| `id` | `bug-<n>`; must match the file name. `buglog.py new` assigns it. |
| `status` | see `.claude/rules/buglog-status.md`; change it with `buglog.py status` |
| `status_updated`, `status_note` | set by `buglog.py status`; a note is required for every status except `fixed` |
| `component` | `controller` · `scheduler` · `worker` · `lib` · `harness` · `apiserver` · `xcecli` · `xceadmin` · `autoscaler` · `build` · `config` · `tooling` |
| `severity` | `data-loss` · `crash` · `hang` · `wrong-result` · `leak` · `security` · `noise` |
| `timestamp` | when the bug was first seen, ISO-8601 UTC, real time (no placeholder `T00:00:00Z`) |
| `last_seen`, `occurrences` | updated by `buglog.py recur` |
| `files`, `tags`, `related` | JSON lists; old status words are not tags (`fixed`, `partially-fixed`, `closed-by-…`) |
| `fix_commits` | short SHAs, required for `fixed` |
| `duplicate_of`, `repro_attempts` | required for `duplicate` / `cannot-reproduce` |
| `legacy` | `true` on records migrated from the old JSON log |
| `gitea_issue`, `gitea_synced_status`, `gitea_synced_hash` | written by `gitea-sync.py` only |
A value of `TODO…` counts as empty: `buglog.py new` writes the template with
`TODO` markers, and `check` lists each one still unfilled.
## Template
`buglog.py new` writes this, with Environment already filled in:
````markdown
---
id: "bug-1872"
status: "open"
status_updated: "2026-09-27"
status_note: "newly logged"
component: "scheduler"
severity: "hang"
timestamp: "2026-09-27T20:05:00Z"
files: []
tags: []
related: []
---
# bug-1872: <symptom in component, ≤ 100 chars>
## Description
**Observed:**
```
<exact error / log / sanitizer text>
```
**Expected:** <…>
**Impact:** <…>
**Found by:** <user report | soak | ASAN | audit | review | test | spec task …>
## Environment
- OS: <distribution + version | "no /etc/os-release" + what it is>; kernel <uname -srm>; <container/VM>
- libc: <name version, from the binary's interpreter>
- Toolchain: gcc <…>, meson <…>
- Build: <asan|debug|release> (<dir>), <sanitizer options>
- Code: <branch> @ <short sha>[-dirty]
- Deployment: <topology>, <worker kind>, <hosts>
## Reproduction
1. <command>
2. <command>
- Frequency: <deterministic | N/M runs, date>
- Evidence: <durable path | "not kept">
## Root cause
<mechanism; since when; scope>
## Fix
<what changed and why it removes the cause>
## Verification
- Test: <name> in <file>: RED before, GREEN after
- Suites: <suite results, profile>
- Live: <soak mode, rounds, result | n/a because …>
## References
- Related: <bug-id (relationship)>, …
- Docs: <path>, …
````
## Example
`.wolf/bugs/bug-1866.md` is a complete record written to this standard
(a worker leak: exact sanitizer text, Environment from `buglog.py env`, a
reproduction on the discovery commit, RED/GREEN verification, fix commit).
## Storage and enforcement
- **Records**: `.wolf/bugs/<id>.md`, one per bug. People and agents read and
edit them directly; keep the `## <Section>` headings.
- **Index**: `.wolf/buglog.json` is generated from the records (for OpenWolf's
hooks, `openwolf bug search` and `jq`). Never edit it; `buglog.py` rewrites
it on every write, and `buglog.py index` after a hand edit.
- **Archive**: `.wolf/buglog-archive.json` keeps the 710 retired OpenWolf hook
auto-detections (template messages such as "Missing guard clause", ids
between bug-002 and bug-1672) verbatim. They are not bug reports and are
not in the index.
- **Migrated records** (`legacy: true`, 2026-09-27) show
`_Not recorded (migrated from buglog.json)._` where the old log had no such
field. They are exempt from the reporting sections; changing one to
`fixed` or `partial` still needs the fix-time sections.
- **Enforcement**:
- `buglog.py status` refuses a status whose requirements the record does
not meet, and writes nothing.
- `buglog.py check <id>|--all` lists every unmet rule.
- The pre-commit hook (`scripts/git-hooks/pre-commit`, installed per clone
with `ln -sf ../../scripts/git-hooks/pre-commit .git/hooks/pre-commit`)
runs `check --staged`: it refuses staged records that miss the standard,
a legacy record hand-changed to fixed/partial without the fix-time
sections, and a stale index.

136
.claude/skills/gitea-sync/SKILL.md
View file

@ -1,6 +1,6 @@
---
name: gitea-sync
description: Sync cc-sdd specs and OpenWolf buglog to Gitea milestones/issues in nexa/xce
description: Sync cc-sdd specs and bug records (.wolf/bugs/*.md) to Gitea milestones/issues in nexa/xce
argument-hint: [--dry-run] [--spec <name>] [--bugs] [--since <date>] [--ids <ids>]
triggers:
- /gitea-sync
@ -11,7 +11,7 @@ triggers:
# Gitea Sync Skill
Sync cc-sdd specs (`.kiro/specs/`) and OpenWolf buglog (`.wolf/buglog.json`) to
Sync cc-sdd specs (`.kiro/specs/`) and bug records (`.wolf/bugs/*.md`) to
Gitea issues in `nexa/xce`. Files are ground truth — Gitea is a read-only mirror.
Sync state (issue numbers) is stored in source files for idempotency.
@ -54,7 +54,7 @@ Keep the script and this procedure in step when changing either.
with `--since`). Needed because bulk triage has bumped `last_seen` /
`status_updated` on most of the buglog, and for a never-synced bug
`--since` only checks `last_seen` — a recently fixed older bug is otherwise
unreachable. Always `--dry-run` first: `--since` alone can match ~1100 bugs.
unreachable. Always `--dry-run` first: `--since` alone can match hundreds of bugs.
### 1. Ensure labels exist
@ -149,9 +149,12 @@ For each spec directory in `.kiro/specs/` (or just `--spec <name>`):
### 3. Sync Bugs
Read `.wolf/buglog.json`. Each entry's `status` field is the source of truth
for open/closed — **never infer it from `fix`**: many entries have a populated
`fix` that says "NOT FIXED", "PENDING" or "PARTIALLY FIXED".
Read the bug records, `.wolf/bugs/<id>.md` (format and rules:
`.claude/skills/buglog/bug-record.md`). Each record's front-matter `status` is
the source of truth for open/closed — **never infer it from the Fix text or
tags**: migrated records often quote "NOT FIXED" or "PENDING" from earlier
stages. `.wolf/buglog-archive.json` (retired hook auto-detections) is never
synced.
**Status → Gitea state / label** (`status` values are defined in
`.claude/rules/buglog-status.md`):
@ -165,114 +168,93 @@ for open/closed — **never infer it from `fix`**: many entries have a populated
| `cannot-reproduce` | closed | `bug:cannot-reproduce` |
| `wontfix` | closed | `bug:wontfix` |
| `duplicate` | closed | `bug:duplicate` |
| missing (hook-created entry) | open | — ; report it in the summary as "needs status" |
| missing / unknown | open | — ; report it in the summary as "needs status" |
An issue carries exactly one status label. When changing it, remove only the
other `bug:<status>` labels and add the new one; never replace the issue's
whole label set, which would wipe labels a person added.
**Shared issues**: several entries can point at one `gitea_issue` (e.g. a
**Shared issues**: several records can point at one `gitea_issue` (e.g. a
user-reported issue whose cause was split into several bugs, like #46). Such
an issue follows its "most open" entry, in this order: `open`,
an issue follows its "most open" record, in this order: `open`,
`investigating`, `partial`, (no status), `cannot-reproduce`, `duplicate`,
`wontfix`, `fixed`.
For each entry (after the `--since` filter):
**Title**: `[<bug-id>] <summary>` — the record's `#` title (max 120 chars).
1. **Skip** if it has `gitea_issue` AND `gitea_synced_status == status`
(already mirrored, nothing changed).
**Body**:
```
**Status:** <status> — <status_note, if any>
**Component:** <component> · **Severity:** <severity>
**Fix commits:** [`<sha>`](<GITEA_URL>/<commit repo>/commit/<sha>), … (if any)
**Duplicate of:** / **Repro attempts:** (if set)
2. **Create issue** for entries without `gitea_issue`:
Use `mcp__gitea__issue_write` method `create`.
## <each record section that has content: Description, Environment,
Reproduction, Root cause, Fix, Verification, References, Legacy notes>
**Title**: `[<bug-id>] <first sentence of error_message>` (max 120 chars),
after dropping any leading status banner such as "OPEN, NOT FIXED --" or
"PRIORITY: LOW --" (status lives in the state and labels).
## Affected Files
- `<file>` for each of `files`
**Body**:
```
**Status:** <status> (<status_note>, if present)
<"Duplicate of <duplicate_of>" / "Repro attempts: <repro_attempts>", if present>
## Tags
`tag1` `tag2` ...
## Error
<error_message>
---
*Synced from `.wolf/bugs/<id>.md`*
```
Commit links point at `GITEA_COMMIT_REPO`, default the `origin` remote's
owner/repo (fixes land on a fork branch before they reach `nexa/xce`).
## Root Cause
<root_cause>
**Labels**: `["bug"]` + the status label, `bug:crash`/`bug:data-loss` from
`severity`, and known tags: `asan`/`ubsan` → `bug:asan`,
`SEGV`/`double-free`/`UAF` → `bug:crash`, `race`/`toctou` → `bug:race`,
`ha`/`raft`/`election`/`failover` → `bug:ha`, `data-loss` → `bug:data-loss`.
## Fix
<fix> (if `fix` is an array, join the elements with blank lines;
"*Not yet fixed*" if empty)
For each record (after the `--since`/`--ids` filter):
## Affected Files
- <file> (split on commas), or each element of `files`
1. **Skip** if it has `gitea_issue`, `gitea_synced_status == status` and
`gitea_synced_hash` equals the hash of the title + body it would write now.
## Tags
`tag1` `tag2` ...
2. **Create issue** for records without `gitea_issue`, then close it if the
state is closed (`state: "closed"` may be ignored on create). No milestone.
## Related Bugs
- <related>
3. **Update changed issues**:
- text changed (hash differs): `update` the issue's title and body, so an
issue created while the bug was open gets its Root cause, Fix,
Verification and fix commits once it is fixed.
- status changed: `update` the state and swap the status label (aggregate
status for shared issues), and post a one-line comment with the new
status, `status_note` and fix commits. This also **reopens** bugs whose
status went back to `open`/`investigating`.
---
*Synced from `.wolf/buglog.json`*
```
**Labels**: `["bug"]` + the status label from the table above. Map known
tags to labels: `asan`/`ubsan` → `bug:asan`, `SEGV`/`double-free`/`UAF` →
`bug:crash`, `race`/`toctou` → `bug:race`, `ha`/`raft`/`election`/`failover`
→ `bug:ha`, `data-loss` → `bug:data-loss`.
**State**: from the table above. `state: "closed"` may be ignored on
`create` — create as open, then `update` to closed if needed.
**Milestone**: None (bugs span specs).
3. **Update changed issues**: for entries WITH `gitea_issue` whose `status`
differs from `gitea_synced_status` (or that have no `gitea_synced_status`
yet), `update` the issue's state and swap its status label per the table
(using the aggregate status for shared issues).
This both closes newly fixed bugs and **reopens** bugs whose status went
back to `open`/`investigating` (e.g. a `cannot-reproduce` bug that recurred).
Post a one-line comment with the new status and `status_note`.
**One-time repair**: issues synced before the `status` field existed were
closed whenever `fix` was non-empty, so entries with `gitea_issue` and no
`gitea_synced_status` whose status is `open`/`partial`/`investigating` are
probably wrongly closed — step 3 reopens them.
4. **Write sync state back** to the buglog entry immediately after each
issue is created/updated (not once at the end — a failure part-way through
must not lose an issue number, or the next run creates a duplicate issue), with the surgical helper —
never re-serialize the whole file (`json.dumps`/`jq` rewrite ~1,200
unrelated lines because entries mix raw UTF-8 and `\uXXXX` escapes):
4. **Write sync state back** to the record's front matter immediately after
each issue is created/updated (a failure part-way through must not lose an
issue number, or the next run creates a duplicate issue):
```bash
python3 scripts/buglog.py set <bug-id> \
gitea_issue=<issue_number> 'gitea_synced_status="<status>"'
python3 scripts/buglog.py set <bug-id> gitea_issue=<n> \
'gitea_synced_status="<status>"' 'gitea_synced_hash="<hash>"'
```
Run it from the repo root. It edits only that entry's lines and refuses to
write anything that no longer parses. For entries without a `status`,
record `gitea_synced_status="unknown"`.
For records without a valid `status`, record `gitea_synced_status="unknown"`.
### 4. Report
Print summary: "Specs: N synced, M skipped | Bugs: X created, Y closed, R reopened, Z skipped"
and list any entries with no `status` (hook-created entries) so they can be triaged.
Print summary: "Specs: N synced, M skipped | Bugs: X created, Y closed, R reopened, F refreshed, Z skipped"
and list any records with no valid `status` so they can be triaged.
## Important Rules
- **Files are ground truth**. Never read Gitea state to update local files.
The gitea_issue/milestone_id fields in local files only store sync mappings.
- **Idempotent**. Items with existing `gitea_issue` or `gitea` state are skipped
unless their phase changed (specs) or `status != gitea_synced_status` (bugs).
unless their phase changed (specs) or their status or text changed (bugs).
- **Labels**: Check existing labels before creating. Use label IDs (not names)
in issue create calls — fetch with `mcp__gitea__label_read` first.
- **Rate limiting**: Batch issue creation — create issues sequentially
(a few per turn) to stay within API limits.
- **Buglog is large** (~1,200 entries, ~27K lines). Always use `--since` filter
for practical runs (e.g., `--since 2026-07-18`). Without `--since`, sync ALL
bugs. Filter with `jq`, don't read the file whole.
- **~400 records have never been synced** (older bugs). A run without
`--since`/`--ids` creates an issue for every one of them; use the filters
unless that is the intent, and always `--dry-run` first.
- **Private repo**: `nexa/xce` is private. MCP auth must have access.
## Gitea MCP Tool Reference

12
.claude/skills/xce-audit-invariants/SKILL.md
View file

@ -269,11 +269,13 @@ several minutes per mode.
## Step 5 — Record and report
- Log every genuinely new root-caused bug with the `buglog` skill:
`python3 scripts/buglog.py add --error ... --root-cause ... --file ...
--tags <component>,... --fix "<what changed>" --status fixed` (or `open` /
`partial` with `--note` for what remains). It assigns the id; don't pick one
by hand. For a violation that an existing entry already covers, update that
entry instead: `scripts/buglog.py status <id> ...` / `note <id> ...`.
`python3 scripts/buglog.py new --summary "<symptom in component>"
--component <component> --severity <s> --binary <failing binary>`, fill in
the record per `.claude/skills/buglog/bug-record.md`, and once the fix is
committed `scripts/buglog.py status <id> fixed --commit <sha>` (or
`partial --note "<what remains>"`). It assigns the id; don't pick one by
hand. For a violation that an existing record already covers, update that
record instead: `scripts/buglog.py status <id> ...` / `note <id> ...`.
- Append durable, non-obvious learnings to `.wolf/cerebrum.md`'s Key
Learnings section (anchor the edit near an existing dated entry to avoid
the "## Key Learnings" heading collision — the file has one per dated

14148
.wolf/buglog-archive.json Normal file
View file

File diff suppressed because it is too large Load diff

22387
.wolf/buglog.json
View file

File diff suppressed because one or more lines are too long

38
.wolf/bugs/bug-006.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-006"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-05-21T08:30:20.719Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c"]
tags: ["md", "double-free", "asan", "ha_connect_to_peer", "endpoint_fini", "integration-test"]
related: []
legacy: true
---
# bug-006: AddressSanitizer: attempting double-free on sctx in ha_connect_to_peer
## Description
AddressSanitizer: attempting double-free on sctx in ha_connect_to_peer
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
endpoint_fini calls FREEUP(ep->data) which frees sctx, then ha_connect_to_peer also calls free(sctx) — double-free at lines 3594-3595 and 3600-3601
## Fix
Set ep->data = NULL before endpoint_fini so FREEUP becomes a no-op; then free(sctx) explicitly in the error paths
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-007.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-007"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-05-21T12:53:13.789Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c"]
tags: ["guard-clause", "c", "heap-use-after-free", "asan", "liveness_poll_fn", "ha_arm_heartbeat_timeout_vtask", "race-condition", "integration-test"]
related: []
legacy: true
---
# bug-007: AddressSanitizer: heap-use-after-free in liveness_poll_fn at ha.c:3293 — reading ctx->cancelled…
## Description
AddressSanitizer: heap-use-after-free in liveness_poll_fn at ha.c:3293 — reading ctx->cancelled after ha_arm_heartbeat_timeout_vtask freed the old ctx
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_arm_heartbeat_timeout_vtask freed the old liveness_poll_ctx_t immediately after setting cancelled=1, but the delaymgr task was already in flight reading ctx->cancelled=0 and then continued to read ctx->ha after free
## Fix
ha_arm_heartbeat_timeout_vtask no longer frees old ctx; liveness_poll_fn now frees ctx when it fires with cancelled=1 (guarded by #ifndef XCE_TESTING to avoid bad-free on stack-allocated test ctxs)
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1013.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1013"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-22T00:00:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/commit.c"]
tags: ["ha", "commit", "raft", "term-safety", "starvation", "commit-ceiling", "optA", "liveness", "catch-up"]
related: ["bug-991", "bug-661", "bug-851"]
legacy: true
---
# bug-1013: Stable HA cluster: committed_lsn pinned at 1 (< 5) while all 5 entries are durable on all 3 nodes…
## Description
Stable HA cluster: committed_lsn pinned at 1 (< 5) while all 5 entries are durable on all 3 nodes — STRICT_TERM live failover gate fails 'committed_lsn>=5'
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha-term-safety task 2.4 added a min(durable-majority, raft_commit_index) ceiling in majority_candidate_locked. raft_commit_index is fed only via the library ACK path (ha_raft_on_repl_ack -> raft_advance_commit -> on_commit_advanced), which advances from match_index[] driven by the asynchronous catch-up streamer and routinely LAGS the durable majority. When delivery goes via catch-up, raft_commit_index stays low and PINS committed_lsn below the safe durable watermark (a redundant, starvation-prone gate). The durable-majority barrier was already current-term-safe under single-writer + the own-term NOOP writable gate, making the ceiling redundant.
## Fix
Option A: removed the raft_commit_index fold from majority_candidate_locked (commit.c). The durable-majority barrier + own-term writable_lsn gate are now authoritative for commit-advance. commit_set_raft_commit_index / ce->raft_commit_index made INERT-ADVISORY (recorded for observability/conformance-oracle only; still fed by on_commit_advanced so the library half keeps a sink). Kept durable self-cap (persist_durable_lsn), persist-before-ACK, and assert(cand<=persist_durable_lsn). Verified live: commit advances 4->5 via durable barrier (commit.c:184); walcompare OK committed prefix byte-identical across 3 nodes up to LSN 5.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-991, bug-661, bug-851

38
.wolf/bugs/bug-1023.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1023"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-23T03:45:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["ha", "raft", "failover", "leader-completeness", "bug-828", "bug-795", "durability", "truncate", "election", "durable_lsn"]
related: ["bug-661", "bug-795", "bug-828", "bug-851", "bug-854", "bug-339"]
legacy: true
---
# bug-1023: Clean single failover (STRICT_TERM) loses committed jobs: safe_committed_lsn=1 from votes despite…
## Description
Clean single failover (STRICT_TERM) loses committed jobs: safe_committed_lsn=1 from votes despite a standby being genuinely durable at LSN 5; walcompare reports committed-prefix divergence at LSN 2 (jobs gone on the new leader).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_enter_real_election() PRE-TRUNCATED the candidate's WAL above ha->last_known_committed_lsn BEFORE the vote. last_known_committed_lsn is the LOCALLY-LEARNED commit watermark, which LAGS the true cluster commit point (the primary commits an entry on a durable majority, then dies before a heartbeat carries that committed LSN to the standby). persist_truncate_wal_above clamps durable_lsn down to the cutoff (clamp_durable_lsn_locked). So a standby durably holding committed entries [committed+1..tip] discarded them, then advertised the stale-low cutoff as its durable floor (ha_candidate_advertise_lsn / ha_vote_self_freshness both read persist_durable_floor). Every voter doing the same collapsed safe-committed-from-votes to the stale watermark; the AUTHORITATIVE post-election truncate (raft_cb_become_leader -> persist_truncate_wal_above(safe_committed_lsn)) then kept only that stale prefix, losing committed jobs. Live trace (DBG828) PROVED the standbys wrote+fdatasync'd LSN 2-5 (durable_lsn=5) and only LOST them at the pre-election truncate clamp — NOT a durability/ack-before-fsync violation. Final bug-828/795 slice.
## Fix
Removed the pre-election persist_truncate_wal_above(last_known_committed_lsn) in ha_enter_real_election (src/controller/ha.c). Raft model: a candidate does NOT truncate its log before campaigning — it keeps its full durable log and advertises its real durable tip so voters holding more committed entries deny a colder candidate and the winner has every committed entry (Leader Completeness). Divergent UNCOMMITTED tails are removed only by the authoritative post-election safe-commit truncate (above the majority-held point) and by the library's per-entry (term,lsn) reconcile (on_truncate_from) during AppendEntries. RED: live STRICT_TERM harness safe=1 + walcompare divergence at LSN 2. GREEN: every run now logs safe_committed_lsn=5 (from 2 granted voters) and walcompare 'committed prefixes identical'. Unit: new test_repl_catchup_durable_floor_survives_failover (test_repl_apply.c) + 177/177 unit GREEN. NOTE: live EXIT still 1 due to a PRE-EXISTING libxce client-failover bug (bug: 'client op fails to reach live primary when a DEAD controller is FIRST in the masters list') surfacing now that the durability bug no longer masks the post-walcompare expect-job/wait-primary step — committed jobs are physically present on disk (walcompare proves it).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-661, bug-795, bug-828, bug-851, bug-854, bug-339

38
.wolf/bugs/bug-1028.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1028"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-23"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c (ha_dispatch_repl_action_for_standby / catch-up streamer on the NEW term)", "config_repl.c (ha_stream_wal_to_standby)", "persist.c (standby apply) -- NOT data loss; committed prefix is byte-identical on disk (walcompare OK)."]
tags: ["failover", "availability", "liveness", "catch-up", "new-primary", "not-serving", "ha-commit-durability", "NOT-FIXED", "FIXED", "catch-up", "ha_stream_wal_to_standby", "persist_entry_term", "logOk"]
related: ["bug-828", "bug-854", "bug-795", "bug-991"]
legacy: true
---
# bug-1028: Post-failover AVAILABILITY: after a clean failover the new primary never becomes available to…
## Description
Post-failover AVAILABILITY: after a clean failover the new primary never becomes available to serve clients (wait-primary times out). New primary logs 'ha_dispatch_repl_action_for_standby: standby[0] lsn=5 < committed_lsn=6 -> stream WAL batches' every ~3s indefinitely; the lagging standby never advances to the new primary's own-term tip, and the client cannot reach a serving primary. Surfaced only AFTER bug-828 was fixed (the committed-job-loss previously masked this by failing the harness earlier at walcompare).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_stream_wal_to_standby (config_repl.c) stamped each streamed entry's term and the AppendEntries prev_log_term from raw on-disk hdr.term (the WAL header), but the standby's logOk (raft_on_append_entries_request) and the library's own send path both source terms from persist_entry_term (the authoritative in-memory per-LSN term index), which maps a legacy-0 entry to WAL_LEGACY_TERM(1). So for a legacy-0 predecessor entry (raw hdr.term==0), the streamer sent prev_log_term=0 while the standby computed its own term as 1 → logOk mismatch → the new-term NOOP (the post-promotion first entry) is perpetually rejected → no WalAck → applied_lsn stuck at pre-promotion value → the heartbeat state report advertises the stale lsn → the primary re-streams [lsn+1..committed] every heartbeat cycle. The committed prefix stays byte-identical because only term metadata diverges, not data. CONFIRMED: baseline walcompare shows committed-prefix DIVERGENCE (term mismatch → CRC-chain breaks on some members); with the fix the committed prefix is IDENTICAL.
## Fix
FIXED (2026-06-23). ha_stream_wal_to_standby gained a const persist_t* p parameter (decl in config_repl.h). All three hdr.term source sites in the streamer body (entry->term, prev_streamed_term on the main path, prev_streamed_term on the OOM-skip path) replaced with p ? persist_entry_term(p, hdr.lsn) : hdr.term (NULL-p path for count-only unit tests). Callers (ha_dispatch_repl_action_for_standby in ha.c, catchup_stream_range in config_repl.c) pass &c->persist. Unit test (test_wal_stream_term_from_index_not_raw_hdr) legacy-appends LSN 1 at term 0 + LSN 2 at term 3; streams standby_lsn=1,committed=2; asserts streamed entry term==3 and prev_log_term==WAL_LEGACY_TERM(1). RED→GREEN proven (Expected 1 Was 0 → PASS). Live: standby converge + committed prefix identical.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-828, bug-854, bug-795, bug-991

38
.wolf/bugs/bug-1060.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1060"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-06-24T12:05:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c (raft_cb_become_leader / safe_committed_lsn)", "src/controller/cluster.c (OPCode_preVoteRequest)"]
tags: ["prevote", "deadlock", "safe_committed_lsn", "truncation", "leader-completeness", "data-loss", "coverage-maneuver", "FIXED"]
related: ["bug-795", "bug-828"]
legacy: true
---
# bug-1060: After soak test coverage maneuver, ctrl-1 and ctrl-2 enter permanent prevote deadlock: ctrl-1…
## Description
After soak test coverage maneuver, ctrl-1 and ctrl-2 enter permanent prevote deadlock: ctrl-1 denies ctrl-2 (freshness: LSN=45 < floor=47), ctrl-2 denies ctrl-1 (log_stale: cand_llt=12 < local_last_log_term=13). Neither can ever win.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ctrl-3 computed safe_committed_lsn=44 from voter peerLastLogLsn (ctrl-3=47, ctrl-2=44 → majority at 44) and truncated committed entries 45-47 (term=12, acked by ctrl-3 itself as standby under ctrl-1). ctrl-3 then wrote leadership NOP at LSN=45 (term=13) and replicated it to ctrl-2. Result: ctrl-1 has LSN=45=JOB_SUBMIT/term=12, ctrl-2 has LSN=45=NOOP/term=13 — divergent WALs. The safe_committed_lsn formula ignores the candidate's own durable knowledge of committed entries, violating Leader Completeness: the new leader must never truncate entries it has that were committed by a prior leader.
## Fix
IMPLEMENTED (2026-06-24): added pure helper ha_promotion_truncate_floor(safe_lsn, durable_tip)=max(safe_lsn,durable_tip) in ha.c/ha.h; raft_cb_become_leader now truncates at max(safe_committed_lsn, persist_durable_floor(&c->persist)) instead of bare safe_committed_lsn. Since the cutoff is never below the leader's own durable tip, persist_truncate_wal_above is a no-op on the leader's durable log — the new leader NEVER truncates entries it holds (Raft §5.4 Leader Completeness). safe_committed_lsn is still the commit lower-bound and promotion-gate HWM (NOT clamped — clamping it would defeat the behind-node fence). The leadership NOP now appends at the true tip (current_lsn) so replication converges peers to the leader's WAL with zero job loss. Unit test test_promotion_never_truncates_own_committed_entries in test_ha_election.c (GREEN). Steering src/controller/CLAUDE.md updated (Leader Completeness section + Post-Election Safe Commit Point + verification checklist). Recovery of an already-diverged cluster still requires restarting the over-truncated node (loses the cut jobs); this fix prevents the divergence.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-795, bug-828

38
.wolf/bugs/bug-1067.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1067"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-24T00:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/config_repl.c", "src/controller/ha.c"]
tags: ["catch-up", "replication", "raft", "ae_reject_stale", "ae_reject_logmismatch", "walCatchupEntry", "cross-term", "FIXED"]
related: ["bug-1028", "bug-991"]
legacy: true
---
# bug-1067: Cross-term WAL catch-up permanently stalls: ctrl-2 stuck at lsn=8, all walCatchupEntry rejected…
## Description
Cross-term WAL catch-up permanently stalls: ctrl-2 stuck at lsn=8, all walCatchupEntry rejected (ae_reject_stale + ae_reject_logmismatch)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
AE term conflation: entry->term is the WAL entry's stored term (e.g. 4 for entries written by ctrl-1 at term=4), but ha_consume_append_entries uses it as the AE header ae_term. For live appends this is correct (leader appends at current term), but for catch-up streaming of historical entries from prior terms, ae_term = entry_term < leader.current_term. The Raft AE stale-term check (if term < s->term → ae_reject_stale) correctly rejects, but it should never fire — the AE header term must always be the leader's current term. There is no separate leader_term field in wal_catchup_entry_t. Consequence: historical entries at entry_term < raft.term are permanently rejected; standby_reported_lsn never advances; streamer retransmits forever.
## Fix
IMPLEMENTED (2026-06-24): separated the AppendEntries HEADER term (leader's CURRENT term) from the per-entry STORED term. Added wire field WalCatchupEntry.leaderCurrentTerm @8 (controller.capnp) + C field wal_catchup_entry_t.leader_current_term (typemap.h, hand-defined struct — capnp generates encode/decode against it). Encoder (request.c wal_entry_to_wire + top container) writes it; decode auto-populates (generated, scalar mapname field). The catch-up streamer ha_stream_wal_to_standby gained a leader_term param (callers pass atomic_load(&c->ha.term)) and stamps entry->leader_current_term=leader_term on every streamed entry. Receiver ha_consume_append_entries now uses ae_term = e->leader_current_term ? e->leader_current_term : entry_term (single-entry branch) — the leader term rides on the sub-entry e by reference through ha_apply_term_entry_via_library, so no extra threading needed. entry->term (@4) remains the per-entry stored term for logOk/on_persist_entry. 0 = unset (legacy/live sender) → falls back to stored term (live appends have entry_term==leader_term, so unchanged). repl.c live paths UNTOUCHED. Tests (test_term_safety_adapter.c): test_intake_crossterm_catchup_uses_leader_current_term (historical entry@term4 streamed by leader@term5 to follower@term5 is ACCEPTED, not stale-rejected) + test_intake_stale_leader_term_still_rejected (leader_current_term=3 < follower term 5 still REJECTED — safety preserved). GREEN: 185 PASS term-safety+config-repl, 6 PASS wal_wire wire round-trip, 80 PASS failover/repl/durability, 0 failures.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1028, bug-991

38
.wolf/bugs/bug-1077.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1077"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "crash"
timestamp: "2026-06-24T17:30:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/xcectx.c"]
tags: ["asan", "use-after-return", "union-type-confusion", "ha", "failover", "ctrlstatus", "libxce", "memory-safety", "regression-61e1391", "VERIFIED"]
related: ["bug-646", "bug-1060", "bug-1067"]
legacy: true
---
# bug-1077: AddressSanitizer: SEGV / stack-use-after-return in free_CtrlStatus (lib/controller.capnp.c:11638)…
## Description
AddressSanitizer: SEGV / stack-use-after-return in free_CtrlStatus (lib/controller.capnp.c:11638) via handle_reply -> free_CtrlStatus_ptr (lib/xcectx.c:91) on `ctrlstatus` after failover; READ derefs the dead-stack `getreply` of xce_get_with_condition (xceapi.c)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ctx->reply.u is an UNTAGGED union shared across opcodes (debugmsg/get/jobid/clusterconf/ctrlstatus alias the same 8 bytes). Commit 61e1391 ('fix bug-1060 bug-1067', unrelated Raft work) added speculative free-guards `if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(&ctx->reply.u.ctrlstatus)` (and the clusterconf twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer) because xce_get_with_condition/xce_resval only NULL the slot on the success path, not the xce_get_req<0 / cbuf_get<0 early-returns. The next ctrlstatus reply (worker thread) sees the foreign non-NULL slot and free_CtrlStatus walks it -> stack-use-after-return SEGV.
## Fix
1) Revert the two unsafe free-guards in handle_reply (untagged union slot can never be safely freed by type) — restores prior MOVE_PTR-overwrite behavior. 2) Clear ctx->reply.u.get = NULL on ALL early-return paths of xce_get_with_condition and xce_resval (kill the dangling-stack-pointer leak at its source; also prevents a late OPCode_get reply writing through the dead stack via handle_get_reply). 3) Defense-in-depth: handle_get_reply drops a reply when ctx->reply.u.get is NULL. Regression test: test_handle_reply_ctrlstatus_does_not_free_foreign_slot (exposes xce_handle_reply, proves a foreign heap value in the shared union is not freed). VERIFIED 2026-06-24 (Opus): re-confirmed 3-layer fix present in working tree; RED proven by re-injecting the ctrlstatus free-guard -> ASan double-free at test_ctrlstatus.c:384 (free_CtrlStatus_ptr at controller.capnp.c:11892), then restored -> GREEN. Full libxce suite 291 Tests / 0 Failures / 0 ASan errors under ASAN_OPTIONS=detect_leaks=0.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-646, bug-1060, bug-1067

42
.wolf/bugs/bug-1095.md Normal file
View file

@ -0,0 +1,42 @@
---
id: "bug-1095"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "unknown"
timestamp: "2026-06-24"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/lib/raft/raft.c"]
tags: ["raft", "vote-reply", "idempotency", "election-safety", "split-brain", "conformance", "needs-library-fix", "bug-851-family", "dedup", "stale-buglog-entry"]
related: ["bug-1096", "bug-851"]
fix_commits: ["b5b7e01"]
legacy: true
gitea_issue: 29
gitea_synced_status: "fixed"
---
# bug-1095: Election Safety violation (two leaders at same term) surfaced by raft-sim-conformance step driver…
## Description
Election Safety violation (two leaders at same term) surfaced by raft-sim-conformance step driver (task 4.2): a duplicated VOTE_RSP grant double-counts a single peer's vote
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
raft_on_vote_reply tallies grants with a COUNTER (s->votes_granted++) and explicitly ignores from_idx ((void)from_idx;), so DuplicateMessage of one peer's grant counts that peer twice and a candidate reaches quorum with a single real vote. Diverges from raft.tla where votesGranted is a SET (idempotent). Proven: 3-node, single peer grant duplicated -> candidate becomes LEADER with votes_granted=2.
## Fix
FIXED (verified 2026-07-18 by validate-impl for raft-sim-conformance): src/lib/raft/raft.c raft_on_vote_reply now has a per-peer vote_replies[from_idx] idempotency guard (dedupes a DuplicateMessage grant) plus a term < s->term DropStaleResponse guard, landed together with bug-1096 in commit b5b7e01 (fix(raft): vote-reply per-peer dedup + stale-term drop). This buglog entry was stale (said NOT FIXED) even though the fix has been live since 2026-06-24 per tasks.md raft-sim-conformance Implementation Notes and bug-1096.fix; corrected here for consistency.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1096, bug-851
- Gitea: #29

38
.wolf/bugs/bug-1096.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1096"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "unknown"
timestamp: "2026-06-24"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/lib/raft/raft.c"]
tags: ["raft", "vote-reply", "stale-term", "election-safety", "split-brain", "conformance", "needs-library-fix", "stale-report", "bug-851-family", "stale-response"]
related: ["bug-1095", "bug-851"]
legacy: true
---
# bug-1096: raft_on_vote_reply counts a stale-term vote-reply (term < currentTerm) toward the current election…
## Description
raft_on_vote_reply counts a stale-term vote-reply (term < currentTerm) toward the current election instead of dropping it
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
No DropStaleResponse: a reply with term < s->term fell through to replies_received++/votes_granted++, corrupting quorum accounting (raft.tla DropStaleResponse discards mterm < currentTerm).
## Fix
FIXED (verified 2026-07-02): src/lib/raft/raft.c:832 has `if (reply_term != s->term)` guard before tallying, with an on_trace("vote_reply_stale_drop", ...) at raft.c:455 — exactly the fix this bug's own second entry described. Landed under raft-sim-conformance / raft-commit-safety (both implemented, 18/18 + 23/23 tasks).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1095, bug-851

39
.wolf/bugs/bug-1112.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1112"
status: "fixed"
status_updated: "2026-09-24"
status_note: "b1f289d2"
component: "lib"
severity: "unknown"
timestamp: "2026-06-24T15:15:03.882281Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/raft/raft_sim.c"]
tags: ["raft-sim-conformance", "link-error", "task-5.3", "undefined-reference", "pre-existing"]
related: []
fix_commits: ["b1f289d2"]
legacy: true
---
# bug-1112: ld: undefined reference to 'raft_sim_scenario_drop_dup_reorder' at test_raft.c:8288 — lib test…
## Description
ld: undefined reference to 'raft_sim_scenario_drop_dup_reorder' at test_raft.c:8288 — lib test binary fails to link
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Commit e445404 (task 5.3 drop/dup/reorder) added the raft.h declaration + the test_sim_5_3 test but FORGOT to add the raft_sim_scenario_drop_dup_reorder implementation to raft_sim.c. Symbol is declared+referenced but never defined, breaking the link for the whole xcelib_test binary.
## Fix
Pre-existing, outside task 5.4 boundary. Surfaced while validating task 5.4 (figure8). Needs task 5.3 implementation added to raft_sim.c. NOT fixed by 5.4.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1128.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1128"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-25T03:20:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/testsuite/test_demotion_convergence.c"]
tags: ["test-flake", "demotion-convergence", "persist", "overlayfs", "ha-convergence-recovery", "quiescence", "wal-truncate", "sandbox", "commit-engine-race", "deterministic", "c"]
related: ["bug-343", "bug-704", "bug-795", "bug-828", "bug-9799", "bug-1060"]
legacy: true
---
# bug-1128: test_demotion_replay_yields_committed_only flakes ~16% on overlayfs: boot recovery over surviving…
## Description
test_demotion_replay_yields_committed_only flakes ~16% on overlayfs: boot recovery over surviving WAL must yield the committed prefix ONLY
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
TWO compounding non-determinism sources, both fixture artifacts not product bugs: (1) overlayfs metadata incoherence — a freshly-opened persist_t fd over a just-ftruncate'd WAL intermittently reads the STALE pre-truncate size, so the cross-context fresh recovery replays the already-removed suffix; (2) DEEPER, fs-independent: the fixture wires a LIVE commit engine and on a single node (n_peers=1) the persist durable-advance callback (fsync worker -> on_durable_advance) commits the whole durable log, so persist_flush asynchronously raced committed_lsn UP from the test-set committed_boundary (5) to the appended tip (8) AFTER the test pinned it; the demotion seam reads committed_lsn at task time -> truncated to 8 not 5 -> current_lsn=9 > committed_boundary+1. DIAG proof: committed_boundary=5 live_committed=8 live_tip=9 on failing runs.
## Fix
Test-only, two parts. (a) Detach the durable-advance commit wake (g_dc.persist.on_durable_advance = NULL) and re-pin committed_lsn = committed_boundary right before driving the demotion task, killing the commit-engine race deterministically (helps BOTH fs paths). (b) Hybrid gate by fs type via static dc_path_is_overlayfs() (statfs f_type == OVERLAYFS_SUPER_MAGIC 0x794c7630, defaults to real-fs path on statfs error): on a real fs keep the strong cross-context fresh boot-recovery assertion (fresh->current_lsn <= committed_boundary+1); on overlayfs skip that fd-incoherent step and assert the LIVE-context truncate effect (g_dc.persist.current_lsn <= committed_boundary+1, coherent under p->lock). Both branches prove the same property. Production persist.c truncate (ftruncate+fdatasync) untouched.
## Verification
Full controller suite 20/20 runs: 0 demotion_replay failures, 0 total failures. Group-only 60/60: 0 failures. Baseline (pre-fix) was 4/20. /tmp confirmed overlayfs (statfs magic 0x794c7630), so the overlayfs branch is the one exercised here.
## References
- Related: bug-343, bug-704, bug-795, bug-828, bug-9799, bug-1060

38
.wolf/bugs/bug-1147.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1147"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-26T03:46:38.904419+00:00"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c:1374 (trim_wal_segments)"]
tags: ["durability", "permanent-gap", "stale-report", "trim", "bug-828", "raft-drive-async", "latent", "catch-up", "wal", "snapshot", "bug-795"]
related: ["bug-795", "bug-828", "bug-1060"]
legacy: true
---
# bug-1147: WAL trim_wal_segments deletes any closed segment regardless of standby catch-up cursor →…
## Description
WAL trim_wal_segments deletes any closed segment regardless of standby catch-up cursor → lagging/empty-rejoin standby loses its needed log range with no snapshot backfill (permanent committed-prefix gap)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
trim_wal_segments bounds deletion ONLY by active_seg_lsn (the currently-open segment): `if (seg_start_lsn < active_seg_lsn) unlink()`. It consults no standby acked_lsn, no committed_lsn, no in-flight snapshot boundary. Strategy-A (cursor-bounded retention) discipline is absent. Strategy-B (entity-bearing snapshot install) is also not live — write_snapshot (persist.c:1247) writes only a 16-byte P0 sentinel, not materialized state. So a standby that lags by more than ~one open segment (down across segment rolls, slow link, >64MB lag, or empty rejoin from cursor=0) hits a log range that exists nowhere and cannot catch up.
## Fix
FIXED (verified 2026-07-02): src/controller/persist.c trim_wal_segments() computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` (min of active segment, all standby acked_lsn, disconnected-slot guards) and only unlinks segments below that floor — comment inline explicitly cites 'bug-1147 guard'. Landed under raft-drive-async task 3.2 (all leaf tasks complete).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-795, bug-828, bug-1060

39
.wolf/bugs/bug-1197.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1197"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-26T00:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/controller/ha.c"]
tags: ["endpoint-quiescence", "c", "dispatch_inflight", "task-3.4a", "tearing_down", "wrong-reference", "raft-drive-async", "task-3.1", "use-after-free", "flag-on-hazard", "deferred", "review-finding"]
related: ["bug-1198"]
fix_commits: ["6e05dc1"]
legacy: true
---
# bug-1197: raft-drive-async finding-1 (FLAG-ON ONLY, dormant): enabling peerctrl_dispatch creates an endpoint…
## Description
raft-drive-async finding-1 (FLAG-ON ONLY, dormant): enabling peerctrl_dispatch creates an endpoint UAF — raft_cb_become_follower runs on a pool worker and endpoint_fini()s all peer vote-eps (frees ep->data) while the single io_worker can concurrently read ep_j->data in peer_ctrl_msg_handler_inline for a same-batch queued event
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Moving peer-control off the sequential io_worker onto the pool co-locates teardown (frees ep->data) and inbound delivery (reads ep->data) on different threads with no unmonitor-before-free ordering. Safe today only because g_drive_async_peerctrl_dispatch_enabled defaults OFF.
## Fix
SUPERSEDED 2026-06-30 by bug-1298 endpoint reference counting (commit 6e05dc1). The pool-reachable teardown-frees-peer-ep UAF class is now governed generally by endpoint_ref/endpoint_unref (refcnt + heap_alloc gate) acquired per ep-bound VTASK_IO in vtask_enq and released in run_task/taskq_item_killer — a queued vtask holds a ref so the io_worker cannot free the ep out from under the dispatcher. Re-validate when raft-drive-async flips g_drive_async_peerctrl_dispatch ON.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1198

38
.wolf/bugs/bug-1198.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1198"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-06-26T00:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["task-3.1", "review-finding", "stale-report", "raft-drive-async", "deferred", "flag-on-hazard", "deadlock-freedom", "liveness-admission"]
related: ["bug-1197"]
legacy: true
---
# bug-1198: raft-drive-async finding-2 (FLAG-ON ONLY, dormant): peer_ctrl_classify tags all peer-control work…
## Description
raft-drive-async finding-2 (FLAG-ON ONLY, dormant): peer_ctrl_classify tags all peer-control work VTASK_LIVENESS, but the CTRL_PRIMARY repl-ack path can reach HA_REPL_SEND_SNAPSHOT->persist_checkpoint (blocking fdatasync) inline — a blocking op under the never-gated liveness reservation breaks deadlock-freedom
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
repl_ack_decode_and_dispatch -> ha_dispatch_repl_action_for_standby can run the snapshot send inline. LIVENESS is only correct once that long op is offloaded off the pool.
## Fix
FIXED/CLOSED (verified 2026-07-02): src/controller/ha.c:5647 carries an explicit in-code 'CLOSED' banner comment — the bug-1197 endpoint_ref/endpoint_unref bracket around the lane dispatch means the repl-ack path's snapshot send is ALWAYS offloaded before peer_ctrl_classify can tag it VTASK_LIVENESS, so bug-1198 cannot recur. Both flags (peerctrl dispatch + snapshot offload) default ON since bug-1376 was fixed.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1197

38
.wolf/bugs/bug-1199.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1199"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "leak"
timestamp: "2026-06-26T00:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/vtaskmgr.c"]
tags: ["raft-drive-async", "task-3.1", "memory-leak", "vtaskmgr", "packet-ownership", "review-finding"]
related: ["bug-1077"]
legacy: true
---
# bug-1199: raft-drive-async task 3.1: peer-control vtask enqueued-but-not-executed at vtaskmgr_fini leaks its…
## Description
raft-drive-async task 3.1: peer-control vtask enqueued-but-not-executed at vtaskmgr_fini leaks its io_packet_t (only the executed entry transfers ownership via iod->pkt=NULL)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
taskq_item_killer called vtask_fini (frees task->data wrapper only), never the carried io_packet_t. The generic killer could not free the pkt unconditionally because the libxce client path (xcectx.c vtask_handle_msg) does NOT clear ep->fsm_in.pkt — its pkt lifetime is the endpoint's, so an unconditional free would double-free.
## Fix
Added int owns_io_pkt to vtask_t (default 0 in vtask_init). taskq_item_killer frees the pkt ONLY when owns_io_pkt && vtype==VTASK_IO && pkt!=NULL. Controller peer_ctrl_dispatch_to_pool sets owns_io_pkt=1. libxce path keeps owns_io_pkt=0 -> unchanged, no double-free. Verified: leak count 808 unchanged vs baseline, 0 double-free under ASan.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1077

38
.wolf/bugs/bug-1210.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1210"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-26"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c", "src/controller/ha.h"]
tags: ["raft-drive-async", "task-3.2", "snapshot", "slow-lane", "UAF", "feature-flag", "bug-1197", "bug-1198", "liveness"]
related: ["bug-1197", "bug-1198", "bug-1147"]
legacy: true
---
# bug-1210: raft-drive-async task 3.2 snapshot slow-lane offload flag g_drive_async_snapshot_offload defaulted…
## Description
raft-drive-async task 3.2 snapshot slow-lane offload flag g_drive_async_snapshot_offload defaulted ON (=1); header doc also said default ON
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Snapshot offload borrows a standby endpoint shared with the io_worker epoll loop (not owned, unlike reconnect_lane_fn). With the flag ON the send runs on a slow-lane thread while io_worker can RDHUP-free that endpoint => cross-thread UAF (bug-1197 family). Defaulting ON would ship that UAF; the borrowed-endpoint lifetime discipline is not established until task 3.4.
## Fix
Default the flag OFF (dormant landing, same protocol as task 3.1 peerctrl dispatch). Inline path runs on io_worker, serialized with teardown -> no overlap. Rewrote both the ha.c definition comment and the ha.h accessor doc to state OFF is the production default and 3.4 is the gate that flips it ON. bug-1198 stays open by design until then.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1197, bug-1198, bug-1147

38
.wolf/bugs/bug-1224.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1224"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-06-26"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/testsuite/test_ha_dispatch_integration.c"]
tags: ["raft-drive-async", "task-3.4b", "test", "flaky", "vtaskmgr_fini", "pthread_cancel", "teardown-race", "deadlock"]
related: ["bug-924"]
legacy: true
---
# bug-1224: raft-drive-async 3.4b integration tests (test_ha_dispatch_integration.c) hang intermittently…
## Description
raft-drive-async 3.4b integration tests (test_ha_dispatch_integration.c) hang intermittently — controller_unit_test never completes (timeout-killed)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Test teardown race: P1/P2/P3 released a parked vtask body (gate_open) then immediately called vtaskmgr_fini, which tears down workers via pthread_cancel. The tests polled a run-counter incremented MID-body, so fini raced the just-released body still unwinding (esp. P2 REQ_BLOCK bodies in vtaskmgr_blocking_release racing blk_lock teardown) → flaky deadlock. slowlane_fini is graceful (join) so the lane side was safe; vtaskmgr_fini (cancel) was the hazard.
## Fix
Add end-of-body completion counters (g_pc_completed on both pc_probe paths incl. the park path; g_block_done at the end of block_task_entry) and poll them to the full expected count AFTER gate_open and BEFORE vtaskmgr_fini, so every pool body has fully returned (worker idle in cbuf_get) before the cancel-based fini. Verified deterministic 5/5 isolated runs + full suite EXIT=0, ASan-clean.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-924

38
.wolf/bugs/bug-1225.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1225"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "unknown"
severity: "unknown"
timestamp: "2026-06-26"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: [".claude/skills/xce-test-ha/run-ha-soak.sh (wait_for_full_convergence)"]
tags: ["ASAN", "baseline-confirmed", "bug-851", "false-negative", "not-a-regression", "raft-drive-async", "reconverge", "soak", "task-4.2", "xce-test-ha"]
related: ["bug-1042", "bug-924"]
legacy: true
---
# bug-1225: /xce-test-ha live failover soak (ASAN build, real fs ~/tmp): FAIL round 4 "cluster did not…
## Description
/xce-test-ha live failover soak (ASAN build, real fs ~/tmp): FAIL round 4 "cluster did not reconverge to 3-of-3 after restarting ctrl-2"
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Pre-existing HA-core instability under 6-round CHAOTIC repeated failover, NOT a raft-drive-async regression. BASELINE-CONFIRMED: the same soak against pre-Phase-3 binaries (commit 21e26a1, release/non-ASAN) ALSO fails at round 4 (baseline symptom: no-new-primary election stall; current symptom: round-4 reconverge + committed-prefix divergence at LSN 9). Both builds preserve committed JOBS (zero job loss in all completed rounds). The committed-prefix divergence is the documented open bug-851/795 family (chaotic-failover silent divergence; separate HA-core spec). raft-drive-async Phase 3 flags are default-OFF (inert); the live trim-guard only preserves more WAL.
## Fix
Not a raft-drive-async bug (baseline-confirmed). 4.2 regression sweep PASSES for this spec: no NEW failure introduced. Pre-existing bug-851 chaotic-failover divergence + bug-1042 reconverge flakiness remain open for a separate HA-core effort.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1042, bug-924

39
.wolf/bugs/bug-1272.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1272"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-30T02:25:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c (election/PreVote path)"]
tags: ["prevote", "ha", "liveness", "chaotic-soak", "environment", "election", "bug-924", "corrected", "7.2", "fdatasync", "raft", "5.3", "in_catch_up"]
related: ["bug-924", "bug-851", "bug-1042", "bug-1060"]
fix_commits: ["bf88681"]
legacy: true
---
# bug-1272: raft-ha-core-safety 7.2 chaotic soak: round-4 'NO new primary within window (possible election…
## Description
raft-ha-core-safety 7.2 chaotic soak: round-4 'NO new primary within window (possible election stall)' after 3 overlapping kills
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
After 3 overlapping kills the only fully-up-to-date node is killed; the two survivors are both restarted-and-lagging (tips 15 and 11; leader was 19). Both hold the full committed prefix [1..11] (safe candidates) but their PreVote rounds repeatedly time out with no grant quorum (prospective terms frozen, never bumped). Distinct from the 5.2 spurious-failover flap (that is fixed and confirmed). Contributors: freshness mutual-denial between lagging survivors, in_catch_up-denies-all-votes (Req 2.5) with no live primary to catch up from, and/or peer vote connection not established between two nodes that only held a control channel to the dead primary.
## Fix
FIXED at unit level by task 5.3 (commit bf88681): gate the in_catch_up PreVote deny in raft_cb_vote_eligible on a LIVE primary (primary_ctrl_fd>=0 && ha_heartbeat_is_fresh), so a catching-up survivor with a dead primary can grant/win a fresh PreVote (logOk preserves Leader Completeness). CORRECTION: this in_catch_up deadlock is a real latent defect (unit RED→GREEN) but is NOT the soak's round-4 blocker. The soak stall is bug-924 (WAL fdatasync→EINVAL wedges a survivor so it stops answering PreVotes; presents as an election stall). 7.2 live acceptance is blocked on that env defect (out of scope; owned by ha-commit-durability), not on consensus.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-924, bug-851, bug-1042, bug-1060

38
.wolf/bugs/bug-1281.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1281"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "unknown"
severity: "crash"
timestamp: "2026-06-30T03:25:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/controller (restart/recovery/election path — exact site unknown", "no backtrace)"]
tags: ["7.2", "iomgr", "gdb", "election", "crash", "5.4", "restart", "recovery", "heisenbug", "accept-hang", "restart-stability", "segfault", "chaotic-soak"]
related: ["bug-1272", "bug-1282", "bug-924"]
legacy: true
---
# bug-1281: Restart-stability bug: a restarted xce_controller becomes unable to participate at chaotic round…
## Description
Restart-stability bug: a restarted xce_controller becomes unable to participate at chaotic round 4. Release build SIGSEGVs ('Segmentation fault (core dumped) --myname ctrl-3'); under gdb the SAME corruption manifests as a WEDGED inbound-accept/iomgr path (last accept 03:42:03 while the election timer keeps firing) so two restarted survivors can't exchange PreVotes -> permanent round-N election stall.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ROOT-CAUSED via gdb SIGQUIT all-thread bt: an ABBA lock-order deadlock in src/lib async dispatch. io_worker (netio.c:672) holds ep->lock then blocks on q->lock (dataq_enq); vdispatcher dataq_deq (dataq.c:65) holds q->lock then blocks on ep->lock via the vtask_eligible->endpoint_is_task_blocked (netio.c:1001) match callback. io_worker dies -> node stops accepting/answering PreVotes -> round-4 stall. In release the same ep/q race corrupts -> SIGSEGV. See bug-1282 (the actual dispatch defect).
## Fix
The crash/hang IS bug-1282 (UPSTREAM raft-drive-async dispatch deadlock). raft-ha-core-safety 7.2 stays blocked until bug-1282 is fixed upstream, then re-run the gate. No raft-ha-core-safety code change.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1272, bug-1282, bug-924

39
.wolf/bugs/bug-1282.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1282"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "crash"
timestamp: "2026-06-30T04:17:08.329Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/lib/dataq.c:65 (dataq_deq holds q->lock across match cb) + src/lib/vtaskmgr.c:255 (vtask_eligible) + src/lib/netio.c:1001 (endpoint_is_task_blocked takes ep->lock) + src/lib/netio.c:672 (io_worker holds ep->lock across vtask_enq->dataq_enq)"]
tags: ["deadlock", "lock-order", "abba", "vtaskmgr", "dataq", "netio", "io_worker", "vdispatcher", "raft-drive-async", "async-io", "upstream", "async-fix", "md"]
related: ["bug-1281"]
fix_commits: ["25e741d"]
legacy: true
---
# bug-1282: ABBA lock-order deadlock under chaotic churn: io_worker (ep->lock then dataq_enq wants q->lock) vs…
## Description
ABBA lock-order deadlock under chaotic churn: io_worker (ep->lock then dataq_enq wants q->lock) vs vdispatcher dataq_deq (q->lock then vtask_eligible->endpoint_is_task_blocked wants ep->lock). Wedges io_worker -> controller stops accepting connections; in release the race corrupts -> SIGSEGV.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
dataq_deq runs its match callback while holding q->lock, and the callback (vtask_eligible->endpoint_is_task_blocked) acquires ep->lock — inverting the ep->lock->q->lock order io_worker uses. Two-thread ABBA deadlock, deterministic under round-4 chaotic churn. Confirmed via gdb all-thread backtrace on two independently-wedged nodes.
## Fix
FIXED (commit 25e741d): promoted endpoint_t.busy from a 1-bit bitfield to _Atomic int; endpoint_block_tasks/endpoint_unblock_tasks/endpoint_is_task_blocked now lockless (release/acquire), removing the ep->lock acquire from the vtask-eligibility path that ran under q->lock. Verified: post-fix gdb soak has ZERO __lll_lock_wait threads + no endpoint_is_task_blocked in any lock path; controller_unit_test all green. Mirrors dispatch_inflight (bug-1197). NOTE: a SEPARATE release-only restart segfault (bug-1283) remains and still blocks 7.2.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1281

39
.wolf/bugs/bug-1286.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1286"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-06-30T05:30:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/controller/cluster.c:251 (myreq_handler) + src/controller/ha.c:3194 (ha_dispatch_gate)"]
tags: ["segfault", "null-deref", "ep-owner", "myreq_handler", "cluster.c", "ha_dispatch_gate", "restart", "7.2", "ROOT-CAUSED"]
related: ["bug-1282", "bug-1281", "bug-1197"]
fix_commits: ["5db734a"]
legacy: true
---
# bug-1286: xce_controller SIGSEGV in ha_dispatch_gate during chaotic round-4 restart — Thread received…
## Description
xce_controller SIGSEGV in ha_dispatch_gate during chaotic round-4 restart — Thread received SIGSEGV at ha.c:3194 (atomic_load &ha->role) with ha=0x3ff0 (garbage). The release-only crash that blocks raft-ha-core-safety 7.2; DISTINCT from the bug-1282 deadlock (which is fixed).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
NULL ep->owner. myreq_handler does `cluster_t* c = (cluster_t*)ep->owner;` then `ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a walCatchupEntry (op 37) vtask while ep->owner==NULL, so c=NULL and &c->ha = NULL+offsetof(cluster_t,ha) = 0x3ff0 — a garbage pointer ha_dispatch_gate dereferences (atomic_load &ha->role, ha.c:3194) -> SIGSEGV. Backtrace (debug -O0 -g3 under gdb): ha_dispatch_gate(ha=0x3ff0,opcode=37) <- myreq_handler cluster.c:251 <- vtask_entry cluster.c:693 <- run_task/vcpu_entry. ep->owner is NULL because the accept/dispatch path runs a request before owner is wired (or owner cleared mid-flight during teardown).
## Fix
FIXED 2026-06-30 (commit 5db734a). NULL-guard `c = ep->owner` in myreq_handler (cluster.c): a chaotic-restart walCatchupEntry (op 37) vtask ran with ep->owner==NULL → &c->ha==0x3ff0 deref crash at ha.c:3194. Guard returns early when owner unset. Verified: round-4 restart SIGSEGV gone; controller_unit_test green.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1282, bug-1281, bug-1197

39
.wolf/bugs/bug-1296.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1296"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-06-30T13:21:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["raft-ha-core-safety", "election-liveness", "prevote-cross-denial", "dueling-candidates", "task-5.6", "chaotic-soak", "round-7-stall"]
related: ["bug-1281", "bug-1282", "bug-1286", "bug-1292"]
fix_commits: ["81cb09a", "74784b2"]
legacy: true
---
# bug-1296: 10-round chaotic soak FAIL at round 7: all 3 nodes in RDHUP→PreVote→timeout→repeat livelock — zero…
## Description
10-round chaotic soak FAIL at round 7: all 3 nodes in RDHUP→PreVote→timeout→repeat livelock — zero pre_vote_reply_in events (not denials, silence). ctrl-3 sent 20+ PreVote rounds with NO reply received; ctrl-1 and ctrl-2 same loop. Rounds 1-6 PASS clean (all elections + job-verification succeeded); round 7 kills ctrl-2 (term=9 primary), restarts ctrl-3 (no convergence wait — chaotic). 2 surviving nodes (ctrl-1, ctrl-3) both start PreVote; neither processes the other's request → mutual timeout → RDHUP fires again → repeat forever.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
TASK 5.6 dueling-candidate PreVote cross-denial — DISTINCT from the earlier PreVote-WINS + real-election split-vote (which had both nodes winning PreVote then deadlocking in RequestVote). Here PreVote itself starves: each node's PreVote handler likely drops/ignores inbound PreVote requests while in its own PreVote round (no reply sent, not even denial). The task 5.6 RDHUP guard (commit 25e741d+) only prevents restarting failover while in CTRL_ELECTING — it does NOT fix the core problem where two nodes enter simultaneous PreVote rounds and each starves the other's reply. The priority stagger (bug-606) helps when 3 nodes are alive (one grants to break symmetry), but with only 2 survivors (the third restarting without convergence in chaotic mode), both deny each other and the stagger is insufficient.
## Fix
FIXED 2026-06-30 (subsumed by tasks 5.6 commit 81cb09a + 5.7 commit 74784b2). The RDHUP->PreVote->timeout liveloc was the same fixed-watchdog resonance as bug covered by 5.7: per-campaign election watchdog re-randomized to [T,2T] (Ongaro 3.4) + don't-preempt-in-flight-election (5.6). CONFIRMED by a 10-round chaotic soak: 10/10 rounds elect a new primary in a single term increment (4->13), zero term escalation, round 7+ (the old liveloc point) all clean.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1281, bug-1282, bug-1286, bug-1292

39
.wolf/bugs/bug-1298.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1298"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "crash"
timestamp: "2026-06-30T13:36:00-07:00"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/netio.c / src/lib/vtaskmgr.c / src/controller/cluster.c"]
tags: ["netio", "vtaskmgr", "use-after-free", "asan", "endpoint-teardown", "concurrency", "raft-ha-core-safety", "soak", "quorum", "bug-1282-family", "FIXED"]
related: ["bug-1282", "bug-1197", "bug-1286"]
fix_commits: ["6e05dc1"]
legacy: true
---
# bug-1298: AddressSanitizer: heap-use-after-free READ size 4 at endpoint_is_task_blocked netio.c:1008 (vtask…
## Description
AddressSanitizer: heap-use-after-free READ size 4 at endpoint_is_task_blocked netio.c:1008 (vtask dispatcher T3) — endpoint freed by endpoint_killer cluster.c:813 (io_worker T2); ctrl-3 ABORTs mid-chaotic-soak round 3→4, dropping the cluster to 1/3 (no quorum) so round 4 'NO new primary within window'
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Teardown-vs-dispatch race: io_worker (T2) tears down a connection on error — handle_endpoint_error → zarray_del → endpoint_killer free()s the endpoint_t — while the vtask dispatcher (T3) is concurrently scanning the dataq (dataq_deq_first_match → vtask_eligible → endpoint_is_task_blocked) and reads ep->busy on that just-freed endpoint. The endpoint is freed while still referenced by a pending vtask in the dataq; nothing drains/guards the ep's queued vtasks before the free, and no refcount keeps it alive across the dispatcher read. Same netio surface as bug-1282 (ABBA, ep->busy made lockless) and the bug-1197 dispatch_inflight precedent; the bug-1282 lockless read removed the (incidental) lock contention so the dispatcher reaches the read faster, exposing the latent UAF.
## Fix
FIXED (commit 6e05dc1): added endpoint_t reference count (refcnt _Atomic + heap_alloc flag). endpoint_get starts refcnt=1 (owner); vtask_enq takes a per-vtask ref before publishing the task to the dispatcher, run_task/taskq_item_killer release it; every heap-ep teardown site drops its owner ref via endpoint_unref (last unref fini+frees). The dispatcher can now read ep->busy on a queued vtask without the io_worker freeing it underneath. Embedded eps (heap_alloc=0) never freed by unref. Validated: 182 unit tests/0 fail; 6-round ASAN chaotic soak ZERO ASan reports, ctrl-3 survives round 4, all rounds elect.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1282, bug-1197, bug-1286

38
.wolf/bugs/bug-1306.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1306"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "unknown"
timestamp: "2026-06-30T14:31:00-07:00"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/hook.c (xce_run_hooks); job.c:603 gate is correct"]
tags: ["ha-commit-durability", "pre-existing", "not-a-regression", "soak", "task-5.9", "committed-job-loss", "commit-durability", "term-boundary-divergence", "catch-up-gap", "failover", "hook-rc-clobber", "ack-before-majority", "data-loss"]
related: ["bug-795", "bug-828", "bug-851", "bug-1298"]
legacy: true
---
# bug-1306: 6-round chaotic-failover soak FAIL: committed_job_loss=4 committed_divergence=7 (rounds 4-6)
## Description
6-round chaotic-failover soak FAIL: committed_job_loss=4 committed_divergence=7 (rounds 4-6). Committed job 0-0-1782829181-1 permanently absent on reachable primary from round 4 on, across 3 elected primaries; rounds 1-3 clean.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
SHARPENED 2026-06-30 (waldump+log bisect of the 10-round soak): NOT the catch-up prefix-gap/entity-snapshot mechanism. xce_run_hooks (src/lib/hook.c:134-157) returns only the LAST-registered hook's rc (rc overwritten each loop iter; break is BEFORE_EVENT-only). The durable-majority gate works: a term-9 leader's job at LSN 30 timed out commit (peers mid-restart, repl_wait_majority_ack -> commit_submit -> -ETIMEDOUT), persist_job_hook_fun returned -ETIMEDOUT, but a later-registered JOB AFTER_EVENT hook (sched/apiserver) returned 0 and CLOBBERED it -> handle_jobsubmit saw hook_rc==0 -> replied 'jobsubmit ok' for an uncommitted entry -> term-10 leader legally overwrote LSN 30 (term-10 NOOP) -> committed-job-loss. Violates the Commit Durability invariant (reply-success => majority-durable). Evidence: all 3 WALs gapless/CRC-intact/full payloads; lost job only on ctrl-1@LSN30 term9 vs term10 NOOP on majority; ctrl-1 log has the WAIT-deadline-expired WARNs and NO 'rejecting job' line.
## Fix
FIXED 2026-06-30 (raft-apply-decoupling item #1 down payment). Extracted the WAL append + majority-commit barrier out of persist_job_hook_fun's WAIT branch into an explicit persist_submit_wal_and_commit(p, jlist); handle_jobsubmit now calls it directly and reads the commit rc DIRECTLY (rejects on <0 before notifying), so the -ETIMEDOUT can no longer be clobbered by a later notify hook's 0 in xce_run_hooks. The JOB_NEW AFTER hook is now notification-only. PROVEN: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code, A/B isolates the durability fix). Unit: +test_job_hook_new_is_notify_only; persist_hooks 19/0, e2e_hooks 3/0, commit_barrier_reply 22/0, commit_durability 38/0. Remaining soak FAIL is committed_divergence (transient catch-up lag, present=2) + a separate round-4 election escalation tail (5.7 class) — neither is job loss.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-795, bug-828, bug-851, bug-1298

38
.wolf/bugs/bug-1307.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1307"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "data-loss"
timestamp: "2026-06-30T15:05:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["data-safety-regression", "task-5.7", "chaotic-soak", "leader-completeness-violation", "confirmed-via-soak", "committed-job-loss", "election-liveness", "bug-1296-followup", "committed-divergence"]
related: ["bug-1296", "bug-1281", "bug-1282", "bug-1286", "bug-851"]
legacy: true
---
# bug-1307: Task 5.7 election-resonance jitter (74784b2) introduced data-safety REGRESSION: 6-round chaotic…
## Description
Task 5.7 election-resonance jitter (74784b2) introduced data-safety REGRESSION: 6-round chaotic soak shows 7 committed divergences (LSNs 8,12,16,19) + 4 committed job losses (job 0-0-1782831451-1). Previous run (pre-5.7, commit 81cb09a) had ZERO divergences and ZERO job losses through 6 rounds. LSN 19 divergence is present=3 (ALL 3 members have different content at same LSN — silent divergence, textbook bug-851). Election liveness improved (all 6 rounds complete vs stalled at round 7), but the jitter defeats the election-safety balance: faster-starting node wins before catching up on committed prefix → Leader Completeness violation.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Task 5.7 jittered watchdog ([T,2T] randomized) lets a less-complete node start and win an election before catching up on the full committed prefix from the dead primary. With the original fixed watchdog, both survivors fired RDHUP simultaneously → cross-denied → the library's priority stagger resolved the tie safely (the fresher node won after deterministic desynchronization). With jitter, one node can fire much earlier (near T) and the other much later (near 2T) → the early node campaigns with an incomplete log → voters grant based on stale view → winner truncates committed entries it never held. The 5.3 in_catch_up release (which drops the vote-deny when the primary is dead) opens a window where a still-catching-up candidate can win — with the new jitter timings this window is hit frequently.
## Fix
CONFIRMED FIXED via live re-soak (2026-07-03): ran the exact reproducing scenario — 6-round chaotic soak (overlapping kill-the-primary, real ext4 at ~/tmp, ASAN controller binary) — that originally showed 7 committed divergences + 4 job losses. This run: 21 jobs submitted, 0 committed_divergence, 0 committed_job_loss, PASS across all 6 rounds (terms 3->9, all 3 nodes rotated through primary). Matches the earlier circumstantial evidence (raft-ha-core-safety Phase 5 fully complete + prior soak-clean sessions) with a fresh, direct confirmation run — no residual doubt.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1296, bug-1281, bug-1282, bug-1286, bug-851

38
.wolf/bugs/bug-1376.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1376"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-07-02T05:10:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["raft-drive-async", "cold-start", "vtask-pool", "self-deadlock"]
related: ["bug-1197", "bug-1298", "bug-1310"]
legacy: true
---
# bug-1376: raft-drive-async: cluster hangs silently (no crash, no ASAN report) during a --force-snapshot soak…
## Description
raft-drive-async: cluster hangs silently (no crash, no ASAN report) during a --force-snapshot soak with g_drive_async_snapshot_offload + g_drive_async_peerctrl_dispatch_enabled both ON: a freshly-wiped empty node cold-starts, self-elects, then goes silent after 'peer_ctrl_dispatch_to_pool: vtask pool not wired -- running inline' is immediately followed by cluster_init_phase2 wiring the pool while raft_cb_become_follower's deferred-teardown vtasks (bug-1197's mechanism) are being enqueued -- the process never logs another line and is eventually killed by the test harness timeout.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
peer_ctrl_msg_handler (ha.c) held ep->dispatch_inflight incremented across the ENTIRE peer_ctrl_dispatch_to_pool call, including its inline fallback (the common cold-start shape when ha->pool is not yet wired). That inline body can reach raft_cb_become_follower, which -- now that cluster_init_phase2 ran synchronously earlier in the SAME call chain and wired the pool -- tears down its own loser vote-ep (the very ep whose reply triggered this call) via the deferred path (vote_ep_teardown_async), whose endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that SAME ep. That count is held only by this exact call, on this exact thread, and cannot reach 0 until the spin itself returns -- a permanent self-deadlock. Confirmed live via a gdb thread backtrace (GDB_WRAP debug seam added to run-ha-soak.sh, launching the target node directly under `gdb --batch` so it becomes ptrace-legal as gdb's own direct child, then SIGINT to gdb once the log went stale for a bounded interrupt + full backtrace): frame `peer_ctrl_dispatch_to_pool` showed pool=0x0 at dispatch time; the same ep pointer appeared in the nested vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on sched_yield().
## Fix
Moved the ep->dispatch_inflight release into peer_ctrl_dispatch_to_pool itself, at the exact right point per exit path: released BEFORE calling the inline body on every fallback path (pool==NULL, OOM, enqueue failure) -- safe because io_worker is single-threaded and the inline path never enqueues anything, so there is no pool-worker race left to protect against once inline execution is committed to -- and only AFTER the (non-blocking) enqueue completes on the success path, preserving the original protection for the genuinely concurrent pool-worker case. New unit test test_vep_self_teardown_does_not_deadlock (test_ha_vep_teardown.c) uses a bounded-timeout pthread_timedjoin_np so a regression fails loud instead of hanging the suite; RED confirmed via a temporary manual revert (Expected 0 Was 110 == ETIMEDOUT) before restoring the fix. Live-verified with both g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled flipped ON: --force-snapshot soak (the exact repro shape) completes without hanging across repeated runs, confirmed real snapshot installs, committed_divergence=0, committed_job_loss=0; a 6-round --chaotic soak with both flags ON also passed clean; full controller suite green throughout. Both flags now default ON in production (raft-drive-async tasks 3.1/3.2 closed).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1197, bug-1298, bug-1310

38
.wolf/bugs/bug-1381.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1381"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-02T00:35:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/config_repl.c"]
tags: ["raft-drive-async", "configpush", "endpoint-lifetime", "uaf-suspect"]
related: ["bug-1197", "bug-1298", "bug-1376"]
legacy: true
---
# bug-1381: raft-drive-async task 3.3 (config-push slow-lane offload) blocked: push_config_transport_only has…
## Description
raft-drive-async task 3.3 (config-push slow-lane offload) blocked: push_config_transport_only has a pre-existing endpoint-lifetime race, independent of the g_drive_async_configpush_offload flag -- it grabs raw endpoint_t* pointers for every registered standby under c->ha.lock, releases the lock, then spawns one pthread per standby (push_thread) that reads/writes those endpoints unprotected for up to CONF_PUSH_TIMEOUT_MS. If a standby RDHUP-disconnects during that window, the io_worker's teardown path (endpoint_killer -> endpoint_unref, the bug-1298 refcount system) can free the endpoint while a push_thread is still using it -- a UAF. This is NOT specific to the lane offload: the transport fan-out spawns its own independent OS threads regardless of which caller (inline drive-thread or lane worker) invokes it, so the documented 'runs inline on io_worker so teardown and push are serialized on one thread' rationale for keeping the flag OFF does not actually hold -- the race exists in the inline path today too.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
push_config_transport_only (config_repl.c ~line 1366) takes c->ha.lock only long enough to copy standby ep pointers into a push_ctx_t array, then releases the lock before pthread_create-ing one push_thread per standby. Those threads read/write the raw endpoint_t* for the full duration of the send + wait_all_with_timeout(CONF_PUSH_TIMEOUT_MS), with no ref taken on the endpoints and no coordination with concurrent teardown (handle_endpoint_error -> zarray_del -> endpoint_killer -> endpoint_unref). Unlike the snapshot-lane fix (bug-1197/bug-1376 session), no endpoint_ref/endpoint_unref bracket protects this fan-out.
## Fix
FIXED. push_config_transport_only (config_repl.c) now takes endpoint_ref() on each standby ep while the pointer is snapshotted under c->ha.lock, mirroring the snapshot-lane fix (bug-1197/bug-1376). Every ref has exactly one matching endpoint_unref(): from push_thread after it finishes using ctx->ep (new close in push_thread itself), or from push_config_transport_only inline if pthread_create fails for that standby, or from a cleanup loop if the threads[] allocation fails entirely. A concurrent RDHUP teardown (endpoint_killer -> endpoint_unref) can now only drop the OWNER reference while a push is in flight -- it can no longer free ep out from under push_thread. New regression test test_configpush_transport_holds_endpoint_ref_across_send (test_config_repl.c) uses a new push_thread_send_hook test seam (same pattern as xce_wal_stream_request_hook) to park a push_thread mid-send, simulate the owner teardown, and assert ep survives with refcnt intact. RED->GREEN verified via temporary revert (Expected 2 Was 1 with the ref-take removed); full controller suite green (230/230) after restore.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1197, bug-1298, bug-1376

38
.wolf/bugs/bug-1386.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1386"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-02T08:15:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/config_repl.c"]
tags: ["raft-drive-async", "snapshot-transfer", "silent-failure", "force-snapshot-soak"]
related: ["bug-1197", "bug-1376", "bug-1381"]
legacy: true
---
# bug-1386: /xce-test-ha --force-snapshot round: after a wiped ctrl-2 rejoins empty and the primary sends a…
## Description
/xce-test-ha --force-snapshot round: after a wiped ctrl-2 rejoins empty and the primary sends a full-checkpoint snapshot, ctrl-2 sometimes never logs a completed install ("HA: snapshot loaded/legacy" line absent) though the cluster still reconverges to 3-of-3 -- OR the cluster fails to reconverge to 3-of-3 at all. Reproduced with g_drive_async_configpush_offload BOTH ON (0/3 pass) and OFF (1/2 pass) on the same freshly-built xce.build binary, so it is independent of that flag -- discovered while validating the bug-1381 configpush flip.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_send_snapshot_to_standby (config_repl.c, ~line 2596) loops over up to 12 65536-byte chunks calling xce_snapshot_chunk_req_raw(ep, &chunk, buf, chunk_len) WITHOUT checking or propagating its return value. xce_snapshot_chunk_req_raw -> endpoint_enq can return -ECONNRESET (e.g. if the standby endpoint is mid-teardown or reconnecting during the empty-rejoin window) but the loop keeps incrementing offset/chunk_index regardless, and the function still logs "transfer complete" and returns rc=0 at the end. Confirmed via ctrl-2 log evidence: primary logs start+complete for a 735546-byte (12-chunk) snapshot within ~2ms, but ctrl-2 only ever receives a handful of leading chunks (2 in one run, 5 in another) before the stream silently stops -- no ERROR on either side.
## Fix
FIXED 2026-07-02 (config_repl.c): ha_send_snapshot_to_standby's chunk loop now captures the return value of xce_snapshot_chunk_req_raw (production path) and xce_snapshot_chunk_hook (XCE_TESTING path) into send_rc, breaks the loop on send_rc != 0, and propagates it as the function's rc instead of silently continuing. The 'transfer complete' INFO log is now strictly gated on rc==0; a failure path now logs a WARN with checkpoint_lsn/total_size/sent-offset/chunk_index/rc for observability. No receive-side timeout/re-request mechanism was added (out of scope, no existing seam) — the existing per-peer snapshot_inflight[] flag already clears unconditionally on send-thread exit (snapshot_lane_fn, ha.c) and the next heartbeat's SEND_SNAPSHOT decision naturally re-triggers a fresh transfer to a peer whose applied_lsn never advanced, so the existing retry cadence recovers once the endpoint is healthy again. New RED->GREEN unit test test_snapshot_transfer_stops_and_propagates_send_failure (test_config_repl.c) injects a mid-loop -ECONNRESET via the existing xce_snapshot_chunk_hook seam (new snaptest_fail_after_hook) and asserts rc == -ECONNRESET and exactly 2 chunks attempted (not the false-'complete' 3). RED confirmed via temporary revert ('Expected -104 Was 0'); GREEN restored, full controller_unit_test suite clean (0 failures across all groups).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1197, bug-1376, bug-1381

38
.wolf/bugs/bug-1389.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1389"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "unknown"
timestamp: "2026-07-02T09:05:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/lib/request.c (suspected: xce_snapshot_chunk_encode_raw capnp Data encode) or src/controller/ha.c (ha_handle_snapshot_chunk per-chunk append)"]
tags: ["raft-drive-async", "snapshot-transfer", "data-corruption", "capnp", "force-snapshot-soak"]
related: ["bug-1386", "bug-847", "bug-1197"]
legacy: true
---
# bug-1389: /xce-test-ha --force-snapshot: after bug-1386's fix (sender now correctly reports success only on…
## Description
/xce-test-ha --force-snapshot: after bug-1386's fix (sender now correctly reports success only on true completion), the force-snapshot soak still FAILs. ctrl-2's log shows all 12 chunks of a 735546-byte snapshot received, then `ha_handle_snapshot_chunk: persist_load_snapshot failed rc=-5` / `handle_snapshot_transfer: chunk 11 rc=-5` (rc=-5=-EIO). ctrl-1 (sender) correctly logs 'transfer complete' both times — the send genuinely succeeded, full byte count. This is NOT bug-1386 (that was a chunk DROP producing a truncated/short transfer). Byte-comparing the primary's source checkpoint file against the exact bytes ctrl-2 assembled into snap.part -> snap-{lsn}.bin (same run, two separate checkpoint LSNs 0x75 and 0x76, both reproduced) shows: identical total size (735546==735546) but DIFFERENT CONTENT — the received file diverges from the sender's file starting exactly at a SNAPSHOT_CHUNK_SIZE (65536-byte) chunk boundary, then the content is a byte-SHIFTED copy of the correct data (same repeating job-frame strings, same occurrence count, just offset) that re-synchronizes with the source at a later offset. The shift direction/magnitude differs between the two independent attempts (-912 bytes resyncing at 262144 vs +4684 bytes resyncing at ~458752) — nondeterministic, not a fixed structural bug.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
NOT YET PINNED to an exact line — root cause is a chunk-CONTENT-corruption bug in the OPCode_snapshotTransfer chunk pipeline, most likely in the capnp raw-Data encode/decode path for the 64KB `payload :Data` field (xce_snapshot_chunk_encode_raw, request.c:630-671, uses the same capn_new_list8/capn_setv8 pattern as bug-847 — 'WAL Catch-Up Streaming Silently Corrupts Payloads at capnp Segment Boundary' — a proven prior case of this exact capnp pattern corrupting raw-byte payloads at a segment boundary). Ruled out as NOT the cause: (1) receiver-side concurrent chunk processing — vtaskmgr's per-endpoint FIFO (endpoint_block_tasks/endpoint_unblock_tasks) guarantees chunk N+1 cannot dispatch until chunk N's ha_handle_snapshot_chunk (including its fopen/fwrite/fclose) fully returns; (2) sender buf-reuse race — xce_snapshot_chunk_encode_raw copies bytes into the capnp segment synchronously before ha_send_snapshot_to_standby's loop re-fread()s the shared buf; (3) fixed-size packet-pool truncation — alloc_packet (netio.c:416-432) mallocs fresh per packet, sized to capn_size(&c), no shared buffer to overflow into. persist_load_snapshot (persist.c:2440 -> load_snapshot at 2325 -> load_snapshot_entities at 2222) is NOT the bug — it correctly rejects the corrupted file via one of its length-sanity -EIO guards (most likely persist.c:2240-2241, the job-frame payload_len-vs-file_size guard); format_version/job_count field-order misparse was considered and ruled out (load_snapshot consumes and validates format_version at persist.c:2399-2413 before ever calling load_snapshot_entities).
## Fix
FIXED 2026-07-02 (ha.c, ha.h). Root cause pinned via live crc32 instrumentation (TRACE logs at both ha_send_snapshot_to_standby's send point and ha_handle_snapshot_chunk's receive point, correlated by snapshot_lsn+chunk_index): every chunk's CONTENT crossed the wire byte-for-byte intact (crc32 matched exactly for all 12 chunks) — the corruption was never in capnp encode/decode/transport at all. The real bug: OPCode_snapshotTransfer chunks of one transfer are dispatched across the vtask pool and are NOT guaranteed to arrive/execute in chunk_index order (live soak log showed arrival order 0,1,3,4,2,5,6,7,8,9,10,11 across cpu#0-3 workers), but ha_handle_snapshot_chunk appended every chunk's bytes to snap.part in ARRIVAL order (fopen "ab"/fwrite) instead of at the chunk's own chunk_offset (a field already carried on the wire but never used for anything). Reordered chunks landed at the wrong file position: same total byte count, wrong content from the first reordered chunk's offset onward — exactly the observed 'byte-shifted, resyncing later, nondeterministic magnitude' signature. Fix: (1) write each chunk via pwrite(fd, payload, len, chunk->chunk_offset) instead of append — order-independent and safe for concurrent writers (positioned writes to disjoint byte ranges of the same fd don't race). (2) Track completion via a new ha_state_t.snapshot_recv_count/snapshot_recv_total pair (guarded by the existing ha->lock) instead of finalizing on 'this call happens to carry is_last' — under out-of-order arrival, is_last can be processed before an earlier chunk; finalize now fires only once every chunk 0..N-1 has actually been written, and snapshot_recv_total is claimed (reset to 0) by whichever thread crosses the completion threshold so a late straggler chunk can never double-finalize. chunk_index==0 still resets both the stale-file unlink and the new counters (Req 4.4 unchanged). New RED->GREEN unit test test_snapshot_recv_out_of_order_chunks_assemble_correctly (test_config_repl.c) replays the exact non-sequential delivery order and asserts byte-identical assembly; RED confirmed via temporary revert (Expected 0 Was -22, job_count field corrupted by the misordered chunk boundary); GREEN restored, full controller suite clean (0 failures). Also added 2 new deterministic full-size (65536-byte) capnp encode/decode round-trip tests (test_wal_wire.c) that definitively cleared the capnp/transport layer before the real cause was found — permanent regression coverage, the prior test only covered a 300-byte payload. New xce_test snapverify/snapdiff subcommands (built as part of this investigation) were instrumental in diagnosing this live. Live-validated: 3/3 --force-snapshot rounds PASS (genuine 'HA: snapshot loaded ... full job/queue entity restoration' installs confirmed, zero divergence/loss) + 6-round --chaotic PASS.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1386, bug-847, bug-1197

39
.wolf/bugs/bug-1396.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1396"
status: "cannot-reproduce"
status_updated: "2026-09-24"
status_note: "extra committed job (20 vs 19) never reproduced after 9318f997 harness hardening"
component: "build"
severity: "unknown"
timestamp: "2026-07-02T14:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/tools/test-commit-durability.sh (harness) or controller job-recovery path (unconfirmed which)"]
tags: ["ha-commit-durability", "low-priority", "cold-restart-all", "job-count-mismatch", "harness-instrumented", "unconfirmed", "related-fix-applied"]
related: ["bug-661", "bug-1422"]
repro_attempts: "2 fresh repro attempts x3 rounds (2026-07-09), 0 reproduced"
legacy: true
---
# bug-1396: ha-commit-durability task 6.1 (cold-restart-all real-fs reproduction…
## Description
ha-commit-durability task 6.1 (cold-restart-all real-fs reproduction, src/tools/test-commit-durability.sh on a genuine ext4 deploy dir, ~/tmp not overlay): run 5 of 5 reported `expect-jobs: expected 19, actual 20` after a full cold-stop-all + cold-start-all cycle — the cluster had ONE MORE committed job than the harness's own ACKED_FILE (deduplicated via `sort -u`) tracked. All 19 tracked job IDs individually verified present (`expect-job` loop all `ok`); the 20th job's identity was not captured (cluster had already been torn down by the script's EXIT trap by the time this was investigated). This is the OPPOSITE of bug-661/795's symptom (loss) — an unexplained EXTRA job, not a missing one. 4 of the other 5 runs (and every prior round within run 5 itself, rounds 1-8) showed correct exact-count durability with zero anomalies.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
- Attempts: 2 fresh repro attempts x3 rounds (2026-07-09), 0 reproduced
## Root cause
NOT INVESTIGATED — deprioritized because it doesn't match the loss scenario task 6.1 was probing for, and reproducing it requires re-running with a 'joblist' capture at the exact expect-jobs failure point (the script doesn't dump the full list on a count mismatch, only the count). Two live hypotheses, unconfirmed: (a) harness-side race — ACKED_FILE's write (submit_n) and run_cold_restart_recovery's read (`committed=$(sort -u ACKED_FILE | grep -c .)`) are not synchronized against a submission whose ack arrives asynchronously after the harness moves on, so a genuinely-committed job simply never got recorded in ACKED_FILE — a test-harness bookkeeping gap, not a controller bug; (b) a genuine job-duplication bug on the controller's recovery/replay path creating an extra job entry with a fresh ID. Also observed in this run's ctrl-1.log: repeated unrelated `handle_jobsubmit fail to lock job 63-15--1-1015808[32751]` errors and one `job not durable (rc=-110) — rejecting submission` for a DIFFERENTLY-FORMATTED job ID (not this harness's `0-0-<timestamp>-0` convention) — almost certainly a stray unrelated process hitting the same cluster ports in the shared sandbox, not this test's own traffic, but flagged here in case it's connected.
## Fix
STILL NOT ROOT-CAUSED for the original exact symptom (successful query, count=20 vs expected=19) — 2 fresh repro attempts (2026-07-09, 3 rounds each, real ext4 fs under ~/tmp) did not reproduce an extra job; both runs showed the tracked count exactly matched after the fix below. HOWEVER, both attempts DID hit a DIFFERENT, related bug in the exact same assert_job_count() check: right after the cold-restart-all election, the aggregate expect-jobs/joblist query itself FAILED outright with 'xce_joblist: Unknown error 10014' / 'could not determine job count' (a query FAILURE, not a wrong-but-successful count) — this is the identical client-side race as bug-1422 (libxce's failover retry loop, client_failover.c, can exhaust its 20-attempt budget in a rapid post-election retry storm and surface a stale E_XCE_STANDBY). Confirmed as a false alarm both times: the very next per-id assert_no_loss check (individual expect-job calls) found every job present. Fixed assert_job_count() with the same bounded-retry pattern used for bug-1422 (3 attempts, 2s settle) — re-run after the fix: 7 passed, 0 failed, clean. This closes the QUERY-FAILURE false-negative class in this check, and makes the harness reliable enough to actually catch the ORIGINAL extra-job symptom if it recurs (previously a query failure and a genuine extra-job count were indistinguishable at a glance; now the retry-exhausted case self-heals and no longer masks a real mismatch). The original 'extra job' phenomenon itself remains open/unconfirmed — needs a longer run (original was 1-of-5 full 8-round soaks) to have a realistic chance of firing again.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-661, bug-1422

38
.wolf/bugs/bug-1400.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1400"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-07-02T14:20:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/testsuite/test.c (cmd_jobsub", "~line 1748)"]
tags: ["stale-report", "low-priority", "test-client-only", "input-validation", "crash", "cmd_jobsub", "xce_test"]
related: []
legacy: true
---
# bug-1400: xce_test cmd_jobsub SEGVs (ASAN) on `jobsub sleep 1` — positional args instead of required getopt…
## Description
xce_test cmd_jobsub SEGVs (ASAN) on `jobsub sleep 1` — positional args instead of required getopt flags (-u/-q/-c). getopt silently ignores unrecognized positional args, leaving submit.command/queue etc. NULL, and something downstream derefs a NULL field.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
CONFIRMED STALE for the reported SEGV: git blame shows NULLSTR guards for submit.user/submit.queue/submit.start_info->cmdline were added 2025-11-05 (commit 41f56fa6), long before this bug was logged 2026-07-02 — 'jobsub sleep 1' getopt-returns -1 immediately on the non-option 'sleep' token, then hits the -EINVAL guard at test.c:1838 before any NULL deref. No SEGV path exists in current code; the buglog entry was stale (fix landed under a different/untracked change, buglog never updated). A REAL, separate defect was found in the same function: submit.start_info (calloc'd test.c:1767) was never freed on any of the 4 early-return validation paths (getopt default case + the 3 NULLSTR checks) — a ~128-byte leak per malformed jobsub invocation.
## Fix
SEGV claim: not applicable, already fixed pre-existing. Leak: refactored cmd_jobsub (test.c) to `goto cleanup` on all 4 early-return validation failures instead of a bare `return -EINVAL`, reusing the existing FREEUP(submit.user/queue/resreq/start_info->cmdline/exdata/start_info) block via a new `cleanup:` label — mirrors the `goto clean` idiom already used in queue_ctrl() (queue.c). Verified: controller ASAN build + full controller_unit_test suite green (0 failures) after the change.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1401.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1401"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "leak"
timestamp: "2026-07-02T14:25:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/controller.capnp.c (decode_CtrlStatus_ptr", "generated codec) or its caller in request.c/xcecli"]
tags: ["asan", "low-priority", "controller-status", "ctrlstatus", "memory-leak"]
related: []
legacy: true
---
# bug-1401: ASAN LeakSanitizer: every ctrlstatus reply leaks memory in decode_CtrlStatus_ptr…
## Description
ASAN LeakSanitizer: every ctrlstatus reply leaks memory in decode_CtrlStatus_ptr (lib/controller.capnp.c:11873) — observed during ha-heartbeat-committed-propagation task 4.1 live validation: 120 bytes/1 alloc after a single ctrlstatus call, 12360 bytes/103 allocs after ~a dozen calls (roughly proportional to call count).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
CONFIRMED via codegraph + generated codec trace: src/include/client.h:119 defined `xce_free_ctrlstatus(s)` as `free_CtrlStatus(s)` — the generated free_CtrlStatus() only frees the FIELDS INSIDE the xce_ctrlstatus_t (clusterName/primaryName/responderName/configChecksum/peers list), matching decode_CtrlStatus(). But the real client reply path (xce_handle_reply -> MOVE_PTR from the decoded RequestReply -> xce_ctrlstatus) populates the struct via decode_CtrlStatus_PTR(), which calloc's the xce_ctrlstatus_t itself (controller.capnp.c:11873) before calling decode_CtrlStatus() to fill it. Only free_CtrlStatus_ptr() (which calls free_CtrlStatus() THEN free()s the struct and nulls the caller's pointer) matches that allocation. free_RequestReply/free_Request/MOVE_PTR were all confirmed symmetric and correct — the bug was isolated to this one macro definition. Existing unit tests never caught it because they call free_CtrlStatus(&out) directly on a stack struct, never exercising the xce_free_ctrlstatus() macro used by real client code.
## Fix
src/include/client.h:119 changed to `#define xce_free_ctrlstatus(s) free_CtrlStatus_ptr(&(s))`. Added regression test test_xce_free_ctrlstatus_frees_the_struct_itself (src/lib/testsuite/test_ctrlstatus.c), which decodes via decode_CtrlStatus_ptr (the real allocation path, not the stack-struct helper the other roundtrip tests use) and frees via the xce_free_ctrlstatus macro. RED confirmed: reverting the fix reproduces the exact reported leak signature (LeakSanitizer: 120 bytes in 1 object, decode_CtrlStatus_ptr calloc). GREEN after fix: 285 lib tests / 0 failures / 0 leaks. Controller ASAN build + controller_unit_test also green post-fix (client.h is shared).
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1406.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1406"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-07-03T08:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/testsuite/test_ha_vep_teardown.c"]
tags: ["testsuite", "controller", "segv", "ssl", "iomgr", "pre-existing", "raft-apply-decoupling"]
related: []
legacy: true
---
# bug-1406: controller_unit_test SIGSEGV (full run, no --run filter): Thread 1 crashes in SSL_CTX_free()…
## Description
controller_unit_test SIGSEGV (full run, no --run filter): Thread 1 crashes in SSL_CTX_free() called from iomgr_fini() during test_ha_vep_teardown.c:271 (test_vep_teardown_runs_after_inflight), which runs at controller_test.c main() TEST(test_ha_vep_teardown). Kills the whole binary before any later-registered TEST() can run, so a bare `controller_unit_test` invocation with no --run filter never reaches a final summary and the exit code is 139 (SIGSEGV), not a Unity pass/fail count.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ROOT-CAUSED 2026-07-09: iomgr_init() (src/lib/netio.c:1168-1184) only ever initialized iomgr_t's `lock` and `epoll_fd` fields -- it never set `ssl_ctx` (or `worker`). Every production caller embeds iomgr_t inside a calloc'd/static context struct (cluster_t.iomgr, xce_ctx_t.iomgr), which masked the gap. test_ha_vep_teardown.c stack-allocates `iomgr_t iomgr;` at 4 call sites (lines 235, 294, 349, 476) WITHOUT a memset -- unlike the sibling libxce tests (test_netio_arm_failure.c:52, test_request_send_propagation.c:34) which explicitly `memset(&iomgr, 0, sizeof(iomgr))` before iomgr_init, establishing the real (undocumented) contract. Left un-zeroed, `ssl_ctx` holds whatever garbage bit pattern occupies that stack slot -- non-deterministic across runs/process history (confirmed intermittent: some full-suite runs exit 0, others SIGSEGV, matching the reported non-strictly-deterministic behavior). iomgr_fini()'s `if (iomgr->ssl_ctx != NULL) SSL_CTX_free(...)` guard then passes on that garbage pointer, and SSL_CTX_free() dereferences a wild address -> SIGSEGV. Proven deterministically with a standalone repro (poisoned an iomgr_t with 0xAB, called the real iomgr_init()+iomgr_fini() from libxce.a) producing an identical backtrace (SSL_CTX_free <- iomgr_fini netio.c:1262 <- caller). gdb backtrace from the original report: SSL_CTX_free (libssl.so.3) <- iomgr_fini (netio.c:1262) <- test_vep_teardown_runs_after_inflight (test_ha_vep_teardown.c:271) <- UnityDefaultTestRun <- test_ha_vep_teardown (test_ha_vep_teardown.c:630) <- main (controller_test.c). Not a double-free/UAF and not a cross-test state leak -- iomgr_set_ssl_ctx() is never called anywhere in test_ha_vep_teardown.c, so no real SSL_CTX ownership is even in play; it is purely uninitialized-memory read.
## Fix
FIXED 2026-07-09: src/lib/netio.c iomgr_init() now explicitly sets `iomgr->ssl_ctx = NULL;` and `iomgr->worker = 0;` before touching lock/epoll_fd, so iomgr_init() fully establishes iomgr_t's invariants itself instead of assuming the caller pre-zeroed the struct -- mirrors the same file's do_endpoint_init(), which already fully initializes every endpoint_t field rather than relying on calloc. Verified safe for all production callers (xcectx.c, myapp.c, cluster.c): every one calls iomgr_set_ssl_ctx() AFTER iomgr_init(), never before, so clearing ssl_ctx in iomgr_init cannot clobber a caller-supplied value. No test file was modified -- the library-level fix protects every current and future caller, including the 4 unguarded stack declarations in test_ha_vep_teardown.c. Verified: (1) standalone ASAN repro poisoning the stack before iomgr_init now shows ssl_ctx == NULL post-init and iomgr_fini() returns cleanly instead of crashing; (2) rebuilt controller_unit_test and ran the full suite (no --run filter) repeatedly under ASAN (detect_leaks=0) with zero SIGSEGVs across all runs -- one full run captured 265 Tests / 0 Failures / 0 Ignored, with all 6 tests in the test_ha_vep_teardown group (including test_vep_teardown_runs_after_inflight) passing their own assertions, not just failing to crash.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

39
.wolf/bugs/bug-1407.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1407"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-03T08:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 5
files: ["src/controller/testsuite/test_config_repl.c"]
tags: ["testsuite", "controller", "commit-engine", "build-environment", "false-positive", "raft-apply-decoupling"]
related: ["bug-924"]
fix_commits: ["3ecb724"]
legacy: true
---
# bug-1407: controller_unit_test: test_commit_advance_retry_clears_push_before_commit_fence FAILS ("Expected…
## Description
controller_unit_test: test_commit_advance_retry_clears_push_before_commit_fence FAILS ("Expected TRUE Was FALSE") when running --run test_config_repl (RUN_TEST site: test_config_repl.c:4008). NOTE: Unity's own printed file:line prefix for this failure is misleading/stale (shows test_repl_unit.c:1543 or controller_test.c:1543 depending on which other suites ran earlier in the same process) -- Unity.TestFile is apparently not re-set per test_xxx(void) group in this harness, only Unity.CurrentTestLineNumber is accurate. The true source is test_config_repl.c.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
DEFINITIVELY ROOT-CAUSED 2026-07-03: this was never a source-code bug. It was a stale/corrupted incremental-build artifact in the main worktree's shared `build/` directory, which had been incrementally `ninja -C build`-rebuilt across 5 sequential tasks (1.1-1.5) without ever a fresh `meson setup`. Proof (four independent measurements, isolated via separate `git worktree add` + fresh `meson setup` + `ninja`, never mutating the main worktree in place): (1) pre-feature commit 17b210c, fresh build: 0 failures, 3/3 runs. (2) commit 9b4b805 (task-1.4 HEAD, IDENTICAL source to the failing main-worktree binary), fresh build in a separate worktree: 0 failures, 3/3 runs. (3) main worktree's stale incrementally-rebuilt `build/`, same 9b4b805 commit: 100% failure, 8/8 runs across two investigation sessions. (4) main worktree after `rm -rf build && meson setup --buildtype=debug -Db_sanitize=address && ninja -C build` (full clean rebuild, same source, same commit): 0 failures, 3/3 runs. This conclusively isolates the failure to the incremental build state, not the code. This is a DIFFERENT gotcha from bug-924 (which is about the `meson test` driver hanging on overlayfs fdatasync) -- this one is `ninja -C build`'s incremental compilation silently producing a wrong/stale binary despite reporting success, likely a header-dependency tracking gap on this overlayfs-backed worktree. The prior 'CLOSED' (task 1.4, 70/70) and 'REOPENED' (task 1.5, 10/10 fail) statuses were BOTH real, contradictory, correct measurements of the SAME source code in DIFFERENT incremental-build states -- neither implementer did anything wrong; the build itself was non-deterministic relative to source content.
## Fix
CLOSED 2026-07-03 (high confidence, this time backed by cross-checked fresh-build isolation, not just an in-place re-run). No source-code fix needed -- `test_commit_advance_retry_clears_push_before_commit_fence` has apparently been correct since at least commit 3ecb724 (pre-feature). ACTION ITEM for all future tasks in this series: do not trust `ninja -C build` incremental results alone when a test result looks surprising or contradicts a very recent prior measurement on the same commit -- rebuild with `rm -rf build && meson setup build --buildtype=debug -Db_sanitize=address && ninja -C build` (or isolate via a throwaway `git worktree add` + fresh setup) before concluding a regression exists. Added to `.wolf/cerebrum.md` Do-Not-Repeat.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-924

38
.wolf/bugs/bug-1408.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1408"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "security"
timestamp: "2026-07-03T21:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/job.c"]
tags: ["raft-apply-decoupling", "task-3.1", "handle_jobsubmit", "array-job-submit", "notification", "commit-barrier", "toctou", "round-2-remediation", "test-longjmp-past-teardown"]
related: []
legacy: true
---
# bug-1408: raft-apply-decoupling task 3.1 round-1 handle_jobsubmit(): when an array job submission has >1…
## Description
raft-apply-decoupling task 3.1 round-1 handle_jobsubmit(): when an array job submission has >1 element and ONE element fails post-commit construction (create_job()/makeJob_() returns NULL, e.g. a jobid collision that slipped past jobid_range_exists() pre-check TOCTOU, or a plain calloc failure), the function returned the construction error IMMEDIATELY after the per-element construct loop, before ever building the AFTER_EVENT notify jlist or firing HOOK_DETAIL_JOB_NEW -- even for SIBLING array elements that constructed successfully and are already durably committed on a majority.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
handle_jobsubmit()'s post-commit construction loop set construct_rc<0 on any failed create_job() call but used continue (not break) to keep constructing remaining siblings correctly -- however, immediately AFTER the loop, "if (construct_rc < 0) { reply error; return; }" fired BEFORE the jlist-build + xce_run_hooks(HOOK_AFTER_EVENT, HOOK_DETAIL_JOB_NEW) block, so a single failed element suppressed notification for every successfully constructed sibling in the same array submission. Both the scheduler and apiserver are purely event-driven off this hook with no periodic reconciliation, so the stranded siblings (real, durable, present in c->jobmgr) were invisible until the next controller restart replayed the WAL.
## Fix
Moved the construct_rc<0 early-return in handle_jobsubmit() (src/controller/job.c) to AFTER the jlist-build + xce_run_hooks(HOOK_AFTER_EVENT, HOOK_DETAIL_JOB_NEW, ...) block instead of before it. The jlist-build loop already only finds jobs that key_find() locates in c->jobmgr, so a failed-to-construct element is naturally and silently absent from jlist with no extra bookkeeping -- the hook now fires unconditionally for every successfully constructed element regardless of a sibling's construct_rc, and the client-facing reply still reports the construct_rc error afterward. Added regression test test_jobsubmit_array_partial_construct_failure_notifies_survivors in src/controller/testsuite/test_commit_barrier_reply.c: drives handle_jobsubmit with a 2-element array on a worker pthread, ACKs element 0's WAL entry, then deterministically reproduces the TOCTOU outcome by inserting a stand-in job directly into c->jobmgr for element 1's jobid (via the existing cbrp_make_job helper) strictly after jobid_range_exists() has already run and strictly before the construction loop can run (element 1's commit is still pending on a withheld ACK) -- makeJob_()'s own key_find() then legitimately returns EEXIST for element 1. A new test hook capture (cbrp_jobnew_capture_fn, registered via xce_add_hook) records every jobid delivered through HOOK_AFTER_EVENT/HOOK_DETAIL_JOB_NEW so the test can assert element 0 was both constructed AND notified despite element 1 failing. RED-phase confirmed against the pre-fix code (job 9201 construction failed with errno=17/EEXIST as designed, and the notification assertion for job 9200 failed as predicted). CORRECTION (round-2 review): under the standard verification invocation (--run test_commit_barrier_reply, the full 25-test group), the failing TEST_ASSERT_TRUE_MESSAGE at test_commit_barrier_reply.c longjmps past this test's own inline cbrp_teardown() call (Unity TEST_ABORT semantics) -- the SAME longjmp-past-teardown pathology already documented for round 1's withheld-commit test, not a clean assertion FAIL. It leaks live persist/commit-engine background threads into the next test's setup, producing a deterministic HANG under the full-group command (reproduced 3/3 times, both threads blocked in futex_do_wait). A clean, fast FAIL was only observed when this test was isolated from the other 24 tests in the group -- that isolated form is what the original write-up described. GREEN after the fix (reordering removes the failure entirely, so this hang mode never triggers in the fixed code): both pre-existing single-element jobsubmit tests and the full 25-test test_commit_barrier_reply group plus the full controller_unit_test binary pass (0 failures) via the standard --run invocation. Hardened afterward: the test now captures its assertion-relevant booleans (cbrp_job_exists/cbrp_jobnew_notified) into locals and calls cbrp_teardown() unconditionally before asserting on the captured locals, so a future regression in this exact test can no longer skip cleanup and hang sibling tests.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1409.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1409"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-07-04T17:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c"]
tags: ["raft-apply-decoupling", "task-6.1", "apply-pump", "busy-loop", "infinite-loop", "commit-resolve-pending", "regression-sweep"]
related: []
legacy: true
---
# bug-1409: test_demotion_convergence hung/spun (18.5M+ log lines, timeout kill) once commit.c's…
## Description
test_demotion_convergence hung/spun (18.5M+ log lines, timeout kill) once commit.c's commit_resolve_pending unconditionally called the new persist_apply_pump_kick
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
apply_pump_drain()'s recheck-then-clear retry loop ignored persist_apply_pump_run()'s return code: on a genuine read failure (e.g. committed_lsn advanced in a test fixture with no real WAL bytes on disk for that LSN, rc=-ENOENT), last_applied_lsn never advances, so the very next re-check inside the SAME drain loop sees identical unresolved work and immediately reclaims the scheduled flag and retries forever -- a tight busy loop with no backoff, hammering disk reads and the logger.
## Fix
apply_pump_drain() now checks persist_apply_pump_run()'s rc immediately after clearing apply_pump_scheduled; a negative rc returns immediately instead of re-checking for more work and reclaiming the flag. The failure is already logged loudly inside persist_apply_pump_run itself. A future, genuinely NEW kick (a later real commit-advance event) retries then, at that event's natural cadence, never in a busy spin.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1410.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1410"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-04T17:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/testsuite/test_apply_pump.c"]
tags: ["raft-apply-decoupling", "task-6.1", "test-only", "va_arg", "hook-system"]
related: []
legacy: true
---
# bug-1410: test_apply_pump's spy hook recorded jobid=0 for HOOK_DETAIL_JOB_UPDATE/HOOK_DETAIL_JOB_NEW instead…
## Description
test_apply_pump's spy hook recorded jobid=0 for HOOK_DETAIL_JOB_UPDATE/HOOK_DETAIL_JOB_NEW instead of the real jobid
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
The hook's per-subscriber `data`/cls pointer (bound at xce_add_hook registration time) is separate from the va_list's own variadic arguments. apply_committed_entry's notify calls pass (cluster_t*, payload) as the actual varargs -- the spy hook read the FIRST va_arg as the payload pointer directly, skipping the leading cluster_t* argument every real dispatch call (job.c, persist.c) always sends first.
## Fix
Added `(void)va_arg(ap, cluster_t*);` before extracting the real payload (xlist_t*/job_t*) in both HOOK_DETAIL_JOB_NEW and HOOK_DETAIL_JOB_UPDATE cases, mirroring persist_job_hook_fun's own established va_arg consumption order.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1411.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1411"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-04T17:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/testsuite/test_apply_pump.c"]
tags: ["raft-apply-decoupling", "task-6.1", "test-only", "asan", "repl-broadcast", "fixture"]
related: []
legacy: true
---
# bug-1411: AddressSanitizer global-buffer-overflow in dataq_enq when test_apply_pump.c called…
## Description
AddressSanitizer global-buffer-overflow in dataq_enq when test_apply_pump.c called persist_submit_job_fact with a real 3-member repl setup
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
repl_broadcast tried to enqueue onto the dummy standby `int` stand-ins (g_tap_hep_a/b, opaque identity handles only) as if they were real endpoint_t* objects, overflowing past the 4-byte int into adjacent globals -- the fixture omitted the repl_broadcast_send_hook interception test_commit_barrier_reply.c's cbrp_setup already uses to keep repl_broadcast from ever dereferencing those dummy pointers.
## Fix
Added the identical repl_broadcast_send_hook = tap_drop_send_hook (a no-op stub returning success) in tap_setup/tap_teardown, mirroring test_commit_barrier_reply.c's cbrp_drop_send_hook exactly.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1412.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1412"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-04T17:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/testsuite/test_apply_pump.c"]
tags: ["raft-apply-decoupling", "task-6.1", "test-only", "comment-terminator", "build-break"]
related: []
legacy: true
---
# bug-1412: test_apply_pump.c failed to compile: 'unknown type name RETAIN', stray bytes, __gnuc_va_list…
## Description
test_apply_pump.c failed to compile: 'unknown type name RETAIN', stray bytes, __gnuc_va_list errors cascading from an unrelated location
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
A doc comment contained the literal substring 'WAL_JOB_*/RETAIN' -- the '*' immediately followed by '/' forms a real C block-comment terminator, closing the file's top doc comment many lines early and turning the rest of the comment text into stray code tokens.
## Fix
Reworded to 'a WAL_JOB_SUBMIT WAIT-mode entry mixed with WAL_JOB_{START,FINISH} RETAIN-mode facts', removing the accidental */ sequence. Lesson: never write '<TOKEN>*/<TOKEN>' inside a C block comment, even when '*' is meant as a wildcard/glob character.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1413.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1413"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-07-04T17:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/testsuite/test_apply_pump.c"]
tags: ["raft-apply-decoupling", "task-6.1", "test-only", "longjmp-past-teardown", "off-by-one", "test-hygiene"]
related: ["bug-1408"]
legacy: true
---
# bug-1413: test_apply_pump's burst-ordering test hung the NEXT test (longjmp-past-teardown) after a wrong…
## Description
test_apply_pump's burst-ordering test hung the NEXT test (longjmp-past-teardown) after a wrong off-by-one assertion (expected lsn_before+3, actual watermark is lsn_before+2 for 3 appended entries) failed
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
A TEST_ASSERT failure mid-test longjmps past the rest of the function body, including the explicit tap_teardown() call at the end -- leaving the fsync thread/tmpdir/hooksys from the failed test alive when the next test's tap_setup() ran, hanging it. Root trigger was a genuine off-by-one in the test's own math (current_lsn before N appends is the FIRST assigned lsn, not one less than it; the watermark after N appends is first_lsn + N - 1, not first_lsn + N).
## Fix
Fixed the off-by-one (first_lsn + 2 == watermark for 3 appends starting at first_lsn). Restructured all three tests in the file to capture every assertion-relevant observable into local variables BEFORE any TEST_ASSERT call, and to call tap_teardown() unconditionally before running any assertion -- the exact pattern already established in this codebase for task 3.1's identical hazard (tasks.md Implementation Notes, task 3.1 round 2/3).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1408

38
.wolf/bugs/bug-1414.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1414"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-04T18:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/job.c", "src/controller/persist.c", "src/include/wal_types.h", "src/include/persist.h", "src/controller/include/cluster.h"]
tags: ["raft-apply-decoupling", "task-6.1", "round-2", "job-state-corruption", "merge-staleness", "deferred-apply", "wal-wire-format", "wal_jobstate_t", "_pad-discriminator"]
related: ["bug-1413"]
legacy: true
---
# bug-1414: raft-apply-decoupling task 6.1 round 1 REJECTED: FINISH (LSN N) then a POST_EXEC-stage FAIL (LSN…
## Description
raft-apply-decoupling task 6.1 round 1 REJECTED: FINISH (LSN N) then a POST_EXEC-stage FAIL (LSN N+1) for the same job, with FAIL received before FINISH applies, corrupts the job to RUN|EXIT|PERR (0x106) instead of the correct DONE|PERR (0x108) once both apply in LSN order -- DONE is silently lost and a stray RUN bit appears.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
record_job_fail_fact (job.c) and record_job_update_fact (postdone) precomputed a FINAL absolute state at RECEIPT time (scratch-copy merge for FAIL, inline backstop+PDONE merge for postdone) and baked it into wal_jobstate_t.state -- correct only when receipt and apply are the same event. Once task 6.1 deferred apply past the commit barrier, a second in-flight fact for the same job (e.g. FINISH) could apply BETWEEN this fact's receipt and its own apply, making the receipt-time snapshot stale by the time apply_committed_entry's generic apply_job_update_transition() did its blind absolute overwrite -- silently clobbering whatever the other fact had just applied.
## Fix
Repurposed wal_jobstate_t._pad (previously always-zero, unused) as a discriminator: _pad==0 still means legacy absolute state (job_ctrl's mark_job_fail/mark_job_update live paths, which mutate+capture under one lock atomically -- no staleness risk, unchanged); a nonzero _pad (WAL_JOB_FAIL_STAGE_FLAG|stage for FAIL, WAL_JOB_UPDATE_POSTDONE_MARK for postdone) marks the entry for apply-time recomputation. apply_committed_entry (persist.c) now dispatches WAL_JOB_FAIL/WAL_JOB_UPDATE on this bit, calling the REAL apply_job_fail_transition() / new apply_job_postdone_transition() (job.c) against the job's ACTUAL current state at apply time instead of a stale baked value. New persist_submit_job_fact_ex(p,type,jobid,state,pad) primitive added; persist_submit_job_fact becomes a pad=0 thin wrapper so zero existing call sites needed changes. Also fixed 2 pre-existing test_commit_barrier_reply.c tests that asserted the old wire format directly (cbrp_count_jobstate_entries reading state bits) -- extended the helper to a _ex variant exposing _pad, updated assertions to check the new marker plus a direct call to the guard core on a scratch job_t. New regression test test_apply_pump_fail_racing_finish_merges_against_current_state in test_apply_pump.c drives FINISH then FAIL through the real receipt handlers with both entries durably appended but unapplied, then a single covering ACK; RED-confirmed byte-for-byte (Expected 264 Was 262) against the reverted round-1 code, GREEN after the fix.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1413

38
.wolf/bugs/bug-1415.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1415"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-04T19:30:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/commit.c (commit_resolve_pending", "commit_resolve/dispatch_continuation)", "src/controller/persist.c (persist_apply_pump_kick", "persist_apply_pump_run)"]
tags: ["raft-apply-decoupling", "task-6.2", "vtaskmgr", "race-condition", "orphaned-registration", "lost-reply", "commit_resolve_pending", "persist_apply_pump_kick", "self-healing"]
related: ["bug-1409"]
legacy: true
---
# bug-1415: raft-apply-decoupling task 6.2 review finding (NOT reproduced under the test suite's pool-less…
## Description
raft-apply-decoupling task 6.2 review finding (NOT reproduced under the test suite's pool-less fixtures, NOT fixed): a real vtaskmgr pool (n_vcpus > 1) can run the commit continuation's dispatch (job_commit_resolve_adapter -> jobsubmit_commit_continuation -> persist_register_pending_apply) and the apply pump's kick (persist_apply_pump_kick -> apply_pump_drain -> persist_apply_pump_run) on two DIFFERENT worker threads with no completion-order guarantee, even though commit_resolve_pending() enqueues them in that order. If the apply-pump task happens to finish first, persist_take_pending_apply() finds nothing registered yet for that lsn (0, correct no-op per its own contract), applies the entry, and advances last_applied_lsn past it -- then when the continuation task finally runs and calls persist_register_pending_apply() for that now-already-applied lsn, the registration is permanently orphaned: the walk only ever revisits [last_applied_lsn+1..committed_lsn], so nothing will ever call persist_take_pending_apply() for that lsn again. Net effect: the job is correctly constructed and visible in c->jobmgr, but the client that submitted it never receives a reply -- silently, forever.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
commit_resolve_pending() (commit.c ~793-835, landed by task 1.2/6.1) resolves every ready continuation via commit_resolve() -> dispatch_continuation(), which for a real pool (ce->pool != NULL) enqueues the continuation onto vtaskmgr via vtask_enq_classified(VTASK_LIVE_BLOCK) and returns WITHOUT waiting for it to run; commit_resolve_pending() then calls persist_apply_pump_kick(ce->p, ce->pool), which enqueues a SECOND, independent VTASK_LIVE_BLOCK task onto the SAME pool. vtaskmgr.c's dispatcher (vdispatcher_entry) hands queued tasks to whichever vcpu worker thread is next idle (pick_idle_vcpu/pick_eligible_task) -- with n_vcpus > 1 (the only realistic production configuration; every test fixture in this spec deliberately never calls commit_set_pool, so ce->pool stays NULL and both dispatches run synchronously inline instead, which is why no test in this suite reproduces the race), two tasks enqueued back-to-back can be picked up by two different idle vcpus at nearly the same time, and their COMPLETION order depends on each task's own runtime (a disk-reading WAL-scan-and-apply walk vs. a calloc+mutex-guarded list insert) -- not their enqueue order. This is a pre-existing structural gap from task 1.2 (async continuation dispatch) + task 6.1 (apply-pump-kick wired into the same choke point, same pool) that had no observable consequence until task 6.2 added a real consumer (persist_take_pending_apply) whose registration can now race the very entry it targets.
## Fix
FIXED round 2 (task 6.2): persist_register_pending_apply() (persist.c) now self-heals option (b) from this entry's own proposed fixes -- before linking a new node onto pending_applies, it checks `lsn <= persist_last_applied_lsn(p)`; if the apply pump's walk has already applied `lsn` (last_applied_lsn already at or past it), `on_applied(ctx, lsn)` fires immediately and synchronously on the CALLING thread instead of being linked for a walk pass that will never revisit that lsn. Race-safe in both interleavings: when the walk has not reached `lsn` yet, the check is false and registration proceeds exactly as before (unchanged behavior, verified against the full existing regression suite); when the walk already passed `lsn`, its own persist_take_pending_apply() call for that lsn already ran and found nothing (that is precisely how we know registration is late), so the immediate inline fire can never double-fire alongside it. New deterministic regression test `test_apply_pump_late_pending_apply_registration_self_heals` (test_apply_pump.c) wires a REAL multi-vcpu vtaskmgr_t (n_vcpus=4) via commit_set_pool -- the one test in this spec to do so -- and forces the exact bad interleaving via a barrier (the test's stand-in commit continuation polls persist_last_applied_lsn() until the apply pump has already applied the lsn, THEN registers), rather than relying on natural timing luck. Verified genuine RED (fails with 'Expected 1 Was 0' against the unfixed check, confirming the hang is real and reproducible) -> GREEN (fires exactly once, resource released exactly once, no orphaned registration) after the fix. Full regression sweep (test_apply_pump, test_commit_barrier_reply, test_apply_reapply_idempotency, test_commit_resolve, test_commit_admission_pool, test_demotion_convergence) shows zero new failures -- the only 2 failures present (test_jobsubmit_returns_before_commit_and_registers_pending_apply, test_jobsubmit_array_mixed_outcome_no_phantom_job_for_failed_elements, both in test_commit_barrier_reply.c) are pre-existing/expected per task 6.1's own Implementation Notes (their assertion text predates task 6's apply pump landing; task 7.3's job to update), confirmed identical before and after this fix by reverting to the pre-round-2 baseline and re-running.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1409

38
.wolf/bugs/bug-1416.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1416"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-05T06:15:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 11
files: ["src/controller/persist.c (persist_verify_apply_correctness / persist_recover", "cluster.c cluster_init_phase2)"]
tags: ["raft-apply-decoupling", "task-7.2", "xce-test-ha", "factstream", "persist_recover", "APPLY_DIVERGENT", "false-positive-candidate", "boot-race", "concurrency", "live-regression-found", "recovery-complete-gate", "myapp-boot-sequencing"]
related: ["bug-1409", "bug-1415"]
legacy: true
---
# bug-1416: [xce-test-ha] FAIL: factstream round 1: cluster did not reconverge to 3-of-3 after restarting…
## Description
[xce-test-ha] FAIL: factstream round 1: cluster did not reconverge to 3-of-3 after restarting ctrl-1 -- root cause: persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip (expected last_applied_lsn=72 new_job_count=23); persist_recover_loud_fence_if_unrecoverable: UNRECOVERABLE, staying alive/non-promotable
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Found live via the new raft-apply-decoupling task 7.2 kill-mid-fact-stream soak scenario (--factstream), reproduced 4/4 runs (ASAN debug build, paced-wave ASAN build, release build, and a cleanup-suppressed debug postmortem run) in this sandboxed environment. cluster_init_phase2() calls cluster_open_listener() (which starts accepting network I/O, including replicated/catch-up WAL traffic from an already-serving primary) BEFORE persist_recover() runs its boot-time apply-correctness verification (task 5.1, Req 8.1/8.2). Under a slow/contended boot (observed: persist_recover firing ~36s after process start instead of the normal ~5s, plausibly due to heavy concurrent nix-develop/test-process CPU contention in this sandbox), a standby's listener can already be live and processing real traffic for tens of seconds before its own persist_recover() call finally executes. persist_verify_apply_correctness() (persist.c ~3597) reads p->last_applied_lsn and live-queries count_jobmgr_entries(p->cluster) AFTER replay_wal completes -- both of these are also mutated by the LIVE apply pump (raft-apply-decoupling task 6.1, wired through commit_resolve_pending) reacting to replicated commit-advance events. If live traffic is already flowing and being applied through the live pump DURING the extended pre-persist_recover window, the live-read 'actual' values used by persist_verify_apply_correctness can reflect state beyond what persist_recover's own single WAL-scan pass computed as 'expected', producing a DIVERGENT verdict even though no real apply defect exists -- confirmed via ctrlstatus probes showing committedLsn identical (79) and no actual data loss across all 3 nodes; the fence engaged correctly per its own fail-loud design (stayed alive, non-promotable, kept retrying) but two of three nodes (including the node that went on to win the next election, term=5, first_boot=1) were fenced at the exact moment persist_recover ran. This matches task 5.1's own documented residual-risk note (tasks.md Implementation Notes, task 5.1 entry): 'the position check is diagnostically tautological for THIS SPECIFIC call site today... this will change once task 6's live apply pump gives expectation and actual state a real chance to diverge across threads/time' -- that predicted future divergence has now materialized under a slow-boot + concurrent-live-traffic combination that only a scenario driving traffic this early/aggressively after cluster boot (the factstream scenario) exercises; existing soak scenarios (chaotic/sequential/force-snapshot) submit far fewer jobs, more slowly, and do not reliably land traffic inside a slow node's pre-recovery window. REFINED after further reproduction (10/10 runs total, across ASAN/release builds and job counts from 8 to 20): the trigger is specifically the RESTART/catch-up path, not the initial multi-node cold boot. A standalone control test (3 nodes, no kill, 20s settle before any traffic) submitted jobs and drove jstart/jfinish/jfail cleanly with zero APPLY_DIVERGENT -- confirming the race needs a node that was DOWN while the cluster kept moving (i.e. an actual failover), not merely a fast cold boot: the returning node's persist_recover()/cluster_init_phase2 vtask must complete while it is ALSO receiving a compressed catch-up replication burst covering everything it missed while down, and that burst's volume is inherent to 'kill mid-fact-stream' (the whole point of the scenario) -- a run-ha-soak.sh-side pacing fix cannot eliminate the trigger the way it did for the (separately real but distinct) initial-boot instance the settle-delay mitigation addresses. Reproduced identically down to 8 fact-stream jobs with 2-job/1s-gap waves, so the trigger threshold is low. This plausibly explains why no prior soak scenario in this spec's history (chaotic/sequential/force-snapshot, all using NJOBS=3 by default, un-paced but small) ever surfaced it: none combine a real kill-during-traffic failover with this much job+fact volume until raft-apply-decoupling task 7.2's factstream scenario. Also independently added test.c's print_jobinfo state_raw hex field (task 7.2) and confirmed via a live probe that apply_job_fail_transition correctly applies EXIT|PERR bits even when job_state_str's RUN-checked-first priority ordering hides them in the pretty-printed state string (job showed 'state: RUN' / 'state_raw: 0x106' after a jstart+jfail-without-jfinish sequence) -- a separate, minor, likely pre-existing (not introduced by raft-apply-decoupling) display-only quirk in job_state_str, also not fixed here (out of boundary), but which the factstream phantom-check was updated to work around by reading state_raw instead of the ambiguous pretty string.
## Fix
FIXED (round 2, raft-apply-decoupling task 7.2 review remediation). Root cause confirmed exactly as hypothesized in the prior entry's candidate #2: persist_verify_apply_correctness() compared a WAL-scan-derived 'expected' snapshot against a LIVE re-read of last_applied_lsn/count_jobmgr_entries(), while TWO live mutation paths could write those same values concurrently -- the live apply pump (persist_apply_pump_run, driven by persist_apply_pump_kick off any of commit.c's 6 committed_lsn-advance sites) and the standby live-replication apply path (ha_apply_single_entry / ha_apply_term_entry_via_library in ha.c). Fixed via a new _Atomic bool persist_t.recovery_complete gate (persist.h/persist.c): persist_apply_pump_kick() now no-ops while the gate is closed; ha_apply_single_entry/ha_apply_term_entry_via_library now skip ONLY the in-memory apply_wal_entry() call while closed (durability write + replication ACK + applied_lsn advance are never gated, preserving the existing persist-before-apply-before-ACK contract and bug-1043's durability/materialization decoupling precedent). persist_recover() closes the gate at entry and reopens it (plus issues one deliberate persist_apply_pump_kick catch-up pass) right before returning, regardless of verdict -- covering any direct unit-test caller. ROUND 2 FINDING during live verification: this alone was insufficient -- a node stuck retrying its own election as a losing/timing-out candidate can receive and durably apply live-replicated entries via the ungated (persist_init default=true) path for tens of seconds BEFORE cluster_init_phase2() ever calls persist_recover() for the first time, reproducing the identical false-DIVERGENT symptom via an earlier entry point that persist_recover()'s own bracket cannot see. Closed by adding persist_mark_recovery_pending(&c.persist) in myapp_run() (myapp.c), called immediately after persist_init()/persist_start() succeed and BEFORE ha_start_election_async()/cluster_run() can process any network traffic -- closing the gate from the earliest safe moment in the real boot sequence. persist_init()'s default stays TRUE (ungated) so none of this spec's ~20 existing unit-test fixtures that construct a persist_t directly (bypassing myapp_run entirely) are affected; persist_mark_recovery_pending is never called outside myapp.c. Verified: live 3-node factstream soak (2 rounds, kill+restart both rounds) now PASSes clean with zero APPLY_DIVERGENT/UNRECOVERABLE across 5 persist_recover() calls (2 initial-boot delayed by the SAME pre-existing election-instability condition that originally exposed the round-2 gap, 2 post-kill restarts, all APPLY_CLEAN); broader controller_unit_test sweep (23 groups incl. test_apply_correctness_verification's 3 real-divergence tests, test_apply_pump/_integration, test_ha_core_safety, test_demotion_convergence) shows zero new failures (only the 2 pre-existing, documented task-6.1/6.2 test_commit_barrier_reply.c failures remain, task 7.3's job). Real-divergence detection independently re-confirmed intact (test_apply_divergent_decision_forces_unconditional_fence, test_apply_divergent_loud_fence_sets_sticky_flag_and_stays_alive, test_real_replay_divergence_sets_unrecoverable_fence all still PASS).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1409, bug-1415

38
.wolf/bugs/bug-1417.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1417"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-07T09:30:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["vote", "raft", "self_idx", "index-mapping", "chaotic-failover", "off-by-one", "ha", "ha_repl_ack_abs_peer_idx", "election"]
related: ["bug-828"]
legacy: true
---
# bug-1417: [xce-test-ha --chaotic] FAIL round 4/10: election stall after killing ctrl-1 — ctrl-2 PreVote wins…
## Description
[xce-test-ha --chaotic] FAIL round 4/10: election stall after killing ctrl-1 — ctrl-2 PreVote wins but real election times out despite ctrl-3 granting RequestVote
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
vote_reply_dispatch passes local votes_idx (compressed, excludes self) directly to raft_on_vote_reply which expects ABSOLUTE peer index (includes self). For self_idx=1 (ctrl-2), ctrl-3's vote (absolute=2, local=1) lands in vote_replies[1]=ctrl-2's own self-slot → treated as duplicate → vote silently dropped. With ctrl-1 dead in chaotic mode, ctrl-2 only has self-vote (1 < quorum 2) → eternal timeout. Self_idx=0 and self_idx=2 are partially/fully immune due to index alignment luck. ha_repl_ack_abs_peer_idx() already exists (ha.c:3779) with correct mapping and passing tests — vote_reply_dispatch just doesn't call it.
## Fix
FIXED (src/controller/ha.c vote_reply_dispatch, ~line 5233): raft_on_vote_reply now receives ha_repl_ack_abs_peer_idx(ha, 1, idx) instead of the raw compressed local idx. ha_repl_ack_abs_peer_idx (ha.c:3779, already existed + unit-tested for the heartbeat-ACK path) performs the exact inverse of raft_cb_send_vote_request send-side remap: votes_idx < self_idx ? votes_idx : votes_idx + 1. ha_record_vote_reply (immediately preceding line) is UNCHANGED -- it correctly wants the compressed idx for ctx->votes[]. Confirmed pre_vote_reply_dispatch needs NO equivalent fix: raft_cb_send_pre_vote_request (ha.c:301) already stores the ABSOLUTE peer_idx in vctx->peer_idx for the PreVote path (explicit code comment there), matching raft_on_pre_vote_reply/pre_vote_replies[] absolute-index expectation -- only the real-vote send path (raft_cb_send_vote_request, ha.c:175) stores compressed. RED->GREEN: new unit test test_bug1417_middle_index_real_vote_from_peer2_not_dropped (test_ha_vote_send.c) drives a REAL 3-node election (self_idx=1) through raft_start_election then a genuine peer-2 grant via vote_reply_msg_handler_for_test (the actual decode/dispatch path, not the xce_vote_reply_hook bypass every other real-vote test in this file uses) -- reproduced Expected 2 Was 1 pre-fix, GREEN post-fix (votes_granted=2, role=RAFT_LEADER). Verified: full 79-group controller_unit_test sweep 0 new failures; 10-round --chaotic xce-test-ha soak PASS (committed_divergence=0, committed_job_loss=0, 18 jobs). Soak surfaced an UNRELATED, pre-existing --chaotic harness bug (run-ha-soak.sh run_chaotic_rounds: DOWN variable is not cleared on a skip-this-round no-primary-to-kill path, so a subsequent round calls start_node on the same already-running node without killing it first -- 6 stacked ctrl-2 processes observed piling up over rounds 5-10) -- confirmed via git diff this script was untouched by this fix, logged separately, NOT fixed here (out of bug-1417 boundary).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-828

41
.wolf/bugs/bug-1418.md Normal file
View file

@ -0,0 +1,41 @@
---
id: "bug-1418"
status: "fixed"
status_updated: "2026-09-24"
status_note: "9b173c9a + bug-1421 + raft-apply-decoupling"
component: "controller"
severity: "hang"
timestamp: "2026-07-09T10:30:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 5
files: ["src/controller/ha.c"]
tags: ["ha", "raft", "apply-divergence", "promotion-gate", "chaotic-failover", "liveness", "term-desync", "resolved-via-upstream"]
related: ["bug-1417", "bug-1419", "bug-1421", "bug-1416"]
fix_commits: ["9b173c9a"]
legacy: true
---
# bug-1418: Final resolution pass over bug-1418 (all 4 prior occurrences): the two contributing factors from…
## Description
Final resolution pass over bug-1418 (all 4 prior occurrences): the two contributing factors from the 2026-07-07T18:20 analysis (term desync + apply-divergence promotion-gate liveness deadlock) are both now resolved, not by the originally-proposed gate relaxation.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Re-audited both factors against the CURRENT code (post raft-apply-decoupling 32/32, post bug-1421): (1) TERM DESYNC — ha_start_prevote_async (ha.c:6715) already computes its prospective term from ha->raft.term + 1, not the stale ha->term the original analysis described; ha.c:1009-1010 explicitly syncs ha->term from the raft term on every relevant transition. This half was already fixed independently of bug-1418's own tracked fix. (2) APPLY-DIVERGENCE LIVENESS DEADLOCK — the promotion gate (ha.c raft_cb_become_leader ~line 1130, and promotion_fence_timeout_fn ~line 876) unconditionally refuses promotion while persist_apply_verdict()==APPLY_DIVERGENT, by deliberate design (raft-apply-decoupling task 5.3, Req 8.4: a divergent local materialization must never be trusted to self-clear via any live/structural condition, including durable_floor>=cluster_hwm — only a fresh persist_recover() replay may clear it). The buglog's own previously-proposed fix (relax the gate to self-apply when durable_floor>=cluster_hwm) would have violated this intentional invariant and risked serving genuinely divergent state to clients. Instead, raft-apply-decoupling's tasks 5.1 (checkpoint_lsn mislabeled from WAL tip instead of the logical applied cursor, causing a false APPLY_DIVERGENT on an ordinary crash-during-checkpoint) and 7.2/bug-1416 (a boot-time TOCTOU where a returning node's persist_recover() replay-verify raced the live apply pump / live-replication apply path, producing a false APPLY_DIVERGENT on the restart/catch-up path specifically — the exact scenario bug-1418's residual occurrence hit: 'apply pump lagged behind WAL tip during kill') closed the false-positive detection paths that were producing spurious APPLY_DIVERGENT verdicts in the first place. A genuine (non-false-positive) divergence still correctly fences forever by design.
## Fix
[Consolidated 2026-08-15: this ID previously had 4 duplicate entries in buglog.json (a pgrep-guard test-harness fix, a partial term-desync analysis, and a stale mid-investigation 'NOT FIXED' snapshot -- all superseded by this final resolution pass, which already accounted for all 3 in its own analysis). Collapsed to this single entry; the other 3 removed.]
No code change to the promotion gate itself — it is intentionally correct and must stay unconditional. The actual fix is the combination of already-shipped upstream work: bug-1421 (AE-reject term-discovery blindspot, this session) + raft-apply-decoupling tasks 5.1/5.3/7.2 (false-APPLY_DIVERGENT elimination, closed prior to this session). Validated: two 40-round soaks today (sequential 123 jobs zero-loss all-3-nodes-led; chaotic 123 jobs zero-committed-divergence zero-job-loss) — an order of magnitude more election churn than the original 10-round soak that first surfaced bug-1418 — produced zero apply-divergent promotion refusals and zero liveness deadlocks. Further corroborated by the 30-round sequential+chaotic regression run immediately after this entry.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1417, bug-1419, bug-1421, bug-1416

47
.wolf/bugs/bug-1419.md Normal file
View file

@ -0,0 +1,47 @@
---
id: "bug-1419"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-07-07T14:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/controller/persist.c (apply-correctness verification) + src/controller/ha.c (PreVote freshness gate) + WAL duplicate-submit mechanism (unlocated)"]
tags: ["ha", "election", "raft", "apply-divergence", "promotion-gate", "chaotic-failover", "liveness", "wal-duplicate-submit", "checkpoint-lsn", "kill-mid-apply"]
related: ["bug-1417", "bug-1060", "bug-845", "bug-1418", "bug-1420"]
fix_commits: ["fe4352f"]
legacy: true
---
# bug-1419: [xce-test-ha --chaotic] genuine, reproducible election-liveness stall (not the bug-1418 harness…
## Description
[xce-test-ha --chaotic] genuine, reproducible election-liveness stall (not the bug-1418 harness process-stacking artifact, which was a separate, now-fixed issue): after 2+ overlapping kills, the surviving 2-node set can permanently deadlock -- neither node can ever become primary again for the rest of the run.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Direct log/WAL analysis (waldump on the affected node's data dir) during a live repro superseded two prior, less-precise theories filed as bug-1418 (a shell-script process-stacking artifact, unrelated and separately fixed; and an unverified ha->term/ha->raft.term "term desync" claim that does not match the code -- raft_begin_pre_vote/ha_start_prevote_async both read s->term == ha->raft.term directly, there is no separate stale-field read on the PreVote path).
ACTUAL mechanism observed: waldump on ctrl-3's WAL (the node with the longest/freshest log after the chaotic kill sequence) showed "duplicate submits: 3 (byte-identical: 0)" -- the SAME jobid appended to the WAL twice with DIFFERING payloads (different subtime, different CRC), e.g. lsn=13 and lsn=14 both jobid=0-0-1783418222-0 but distinct subtime/PCRC. This is NOT bug-845's mechanism (that fix, landed 2026-06-23, was for byte-IDENTICAL double-hook-registration dupes and is confirmed still working -- these are non-identical). count_jobmgr_entries() legitimately sees these as 2 distinct job-table mutations (or the entity count otherwise fails to match the WAL-replay-derived expectation), so persist_recover()'s post-replay apply-correctness check (task 5.1, Req 8.1/8.2) correctly fires APPLY_DIVERGENT -> the loud, sticky, non-promotable fence (raft-apply-decoupling Req 8.3/8.4) -- this fencing behavior is WORKING AS DESIGNED, not itself a bug.
The resulting LIVENESS bug: with the fenced node holding the only sufficiently-fresh (longest) log, the other surviving node can never win the PreVote freshness gate against it (correctly, per Raft/PreVote semantics -- it really is behind), while the fenced node itself can win Raft elections (quorum_reached/became_leader observed at terms 27/30/32 in logs) but its own raft_cb_become_leader promotion gate refuses CTRL_PRIMARY every time (apply_divergent=1). With the third node dead (killed by the chaotic driver) and never restarted long enough to catch up, NO node can ever become primary again -- a genuine, reproducible liveness deadlock, distinct from (and downstream of) the WAL duplicate-submit root cause. Confirmed via direct log grep across ctrl-1.log/ctrl-3.log at the exact stall timestamps (09:57:16 divergence detected, repeating "PreVote DENIED ... reason=freshness" and "promotion REFUSED ... apply_divergent=1" through 10:05+); NOT root-caused via speculation.
## Fix
FIXED (2026-07-07). Layer (1), the WAL duplicate-non-identical-submit mechanism: raft-apply-decoupling task 3.1 deferred job/entity construction in c->jobmgr until AFTER commit (apply_committed_entry, post-commit apply pump), but jobid_range_exists() -- the sole dup guard in handle_jobsubmit -- only checks c->jobmgr. A submission that is durably WAL-appended and awaiting its commit outcome does not yet exist in c->jobmgr, so a client-side retry for the SAME jobid (XCECLI_FAILOVER_CALL/xce_failover_classify's idempotent-replay contract resends on a transport error or failover-mid-flight) sailed straight past the guard and appended a second, non-identical WAL_JOB_SUBMIT entry (different subtime/CRC) for the same jobid -- exactly the 'duplicate submits: N (byte-identical: 0)' signature waldump showed. Fix: src/controller/job.c adds a process-lifetime in-flight-jobid-range registry (s_inflight_submits, guarded by s_inflight_submits_lock) that handle_jobsubmit atomically check-and-claims (inflight_range_try_add) immediately after the existing jobid_range_exists check, and releases (inflight_range_remove) from jobsubmit_async_ctx_release. Regression test test_jobsubmit_resubmit_before_resolution_rejected_no_dup_wal_entry added to test_commit_barrier_reply.c (RED->GREEN confirmed). Committed fe4352f.
Layer (2), the PreVote-freshness/apply-divergence liveness deadlock: initially framed as needing a fundamental safety/liveness design tradeoff among 3 unpalatable options (force-promote a divergent node / truncate committed-but-unapplied entries / relax the freshness gate) -- this framing was WRONG, corrected after user pushback ("kill-mid-apply should be recoverable per Raft protocol", "I do not think break FSM invariant is a good idea"). Root-caused instead to a concrete, scoped bug: persist_checkpoint() (persist.c:2261-2263, now fixed) stamped p->checkpoint_lsn from p->current_lsn (the physical WAL append tip) instead of p->last_applied_lsn (the true 'jobmgr reflects up through here' cursor, persist.h:344-363 documents these as DISTINCT once apply is decoupled from commit). write_snapshot() serializes live c->jobmgr, so a checkpoint taken while the async apply pump lags behind the WAL tip mislabels the snapshot: on restart, last_applied_lsn is re-seeded FROM checkpoint_lsn and replay resumes strictly above it, silently skipping committed-but-not-yet-applied entries that jobmgr never actually received -- the real mechanism behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence. Confirmed apply_committed_entry() is fully deterministic/idempotent (rules out non-determinism) and write_snapshot() is crash-atomic at the file level (temp+fdatasync+rename, rules out torn-file corruption) -- the bug is purely a mislabeled watermark, not a structural divergence. Fix: persist_checkpoint() now reads checkpoint_lsn from last_applied_lsn and holds apply_pump_lock across the label-capture + write_snapshot() call so the label and the jobmgr state it describes are captured atomically w.r.t. the live apply pump (the pump only ever moves last_applied_lsn/jobmgr forward, so freezing it here rules out the label ever claiming more than jobmgr actually holds). test_checkpoint_sets_checkpoint_lsn updated (now simulates apply catch-up explicitly, since a cluster-less unit-test persist_t never runs the apply pump); new regression test test_checkpoint_lsn_tracks_last_applied_not_current_lsn added to test_persist_checkpoint.c (RED confirmed pre-fix: checkpoint_lsn took the WAL tip=7 instead of the simulated applied cursor=2; GREEN after). Full 78-group/782-test controller_unit_test sweep: 0 failures.
ADDENDUM (2026-07-08, post-fix re-soak): a fresh chaotic soak on a rebuilt binary (both layers included) cleared rounds 1-4 (previously stalled at round 4) but still hit a fresh APPLY_DIVERGENT fence at round 5. Deep dive (waldump + log correlation + code trace) proved BOTH layer 1 and layer 2 are working correctly and are NOT the cause: added test_jobsubmit_resubmit_after_full_resolution_rejected_no_second_job (test_commit_barrier_reply.c) proving the post-resolution dedup window is also airtight (-EEXIST, no second WAL entry) -- both known dedup windows are provably race-free. The round-5 duplicate jobid was traced instead to a TEST-HARNESS-ONLY defect: xce_test (test.c:1868) hardcoded idgen(&idctx, 0, 0, ...) (clusterId=apiserverId=0) and run-ha-soak.sh's submit_jobs() spawns a fresh xce_test process per retry attempt, resetting idgen's static idctx.sequence to 0 each time -- two unrelated job submissions from two fresh processes landing in the same wall-clock second collide on an identical jobid (ts+tag+seq all equal). Confirmed NOT a production risk: apiserver's GenerateJobID (clib.go) holds one process-lifetime idgen_ctx_t via sync.Once and each deployed apiserver instance has its own distinct apiserverId, so config.h's jobid bit-layout (cluster/apiserver bits) already partitions production's namespace. Fixed by folding getpid() into the harness's apiserverId (test.c:1868) so cross-process collisions become structurally impossible there too; filed as bug-1420. bug-1419 itself (both production layers) stands as correctly fixed -- this addendum exists so a future session doesn't misread the round-5 re-soak stall as evidence layer 1/2 are insufficient.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1417, bug-1060, bug-845, bug-1418, bug-1420

38
.wolf/bugs/bug-1420.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1420"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-08T04:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/testsuite/test.c"]
tags: ["test-harness", "jobid", "idgen", "chaotic-soak", "false-positive"]
related: ["bug-1419"]
legacy: true
---
# bug-1420: xce_test CLI harness generates colliding jobids across separate process invocations within the…
## Description
xce_test CLI harness generates colliding jobids across separate process invocations within the same wall-clock second, surfacing as a spurious APPLY_DIVERGENT fence in chaotic HA soak (misattributed at first to bug-1419's controller-side dedup logic).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
idgen() (src/lib/idgen.c) derives jobid from a per-context {timestamp (1s resolution), sequence (reset to 0 whenever the second rolls over or the context is freshly initialized)} pair, plus caller-supplied clusterId/apiserverId bits (src/include/config.h: 6+4 bits respectively) that are meant to partition the jobid namespace across independent generators. xce_test's cmd_jsub called idgen(&idctx, 0, 0, arsize, ...) -- both partition fields hardcoded to 0 -- so every xce_test process shares the SAME (degenerate, unpartitioned) jobid space. run-ha-soak.sh's submit_jobs() (run-ha-soak.sh:446) spawns a fresh xce_test process per submit/retry attempt (`printf script | xtest`, xtest() = `timeout 60 "$XT"`), and idgen_ctx_init() resets sequence=0 on every fresh process start. Two unrelated job submissions from two different xce_test process launches landing in the same wall-clock second therefore compute an IDENTICAL jobid (ts, tag, seq all equal) -- a genuine, reproducible collision between logically distinct jobs, not a retry of the same one. This is NOT a production defect: apiserver's GenerateJobID (src/frontend/apiserver/backend/clib.go) initializes its idgen_ctx_t exactly once per process via sync.Once (sequence only resets on an actual apiserver restart, far rarer than per-request), and each deployed apiserver instance is assigned its own distinct apiserverId, so config.h's bit layout already partitions the production jobid namespace across instances even in the rare restart-collision case.
## Fix
Folded getpid() into the harness's apiserverId at the idgen() call site (test.c:1868): idgen(&idctx, 0, (size_t)((uint64_t)getpid() & MAX_APISERVER_ID), arsize, &(jobsubmit.jobid)) -- gives every xce_test process invocation its own slice of the same namespace partitioning production already relies on, making a same-second cross-process collision structurally impossible. Mechanical, test-tool-only change; no production code touched. Verified via clean rebuild of both xce_test and controller_unit_test targets (no coupling, no regressions).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1419

38
.wolf/bugs/bug-1421.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1421"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-08T06:55:15.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c"]
tags: ["ha", "raft", "term-desync", "liveness", "election-regression", "ae-reject", "replication"]
related: ["bug-1417", "bug-1418", "bug-851"]
legacy: true
---
# bug-1421: Sequential HA soak regression: no primary elected at term>8 by round 6 (initially suspected to…
## Description
Sequential HA soak regression: no primary elected at term>8 by round 6 (initially suspected to stem from the bug-1417 fix); root-caused 2026-07-09 as the primary never discovering a follower's higher term via AppendEntries-reject responses
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_raft_on_repl_ack only routes AE-reply term-adoption for success=1 ACKs; AE rejects (success=0) never reach raft_on_append_entries_reply. That function additionally drops any reply with reply_term != s->term (raft.c:832-833, stale-term drop) before it can adopt a higher term. Net effect: once a follower is ahead in term (e.g. it steps down from PreVote to CTRL_STANDBY at a higher term while an existing primary keeps campaigning/replicating at its old term), every AppendEntries the stale primary sends is rejected with ae_reject_stale, the primary never learns the higher term, and it can never regain write-quorum (needs 2/3, only self-vote counts) — a permanent livelock. First surfaced as an election regression (sequential soak: no primary elected at term>8 by round 6, chaotic stalled at round 5) suspected to be caused by the bug-1417 vote-index-mapping fix; re-reproduced 2026-07-09 as a round-4 sequential stall (primary ctrl-2 term=6, follower ctrl-3 term=7, 490 ae_reject_stale events, primary writable_lsn stuck at 17), which pinned the exact mechanism above.
## Fix
Added two fallback term-discovery vectors that fire regardless of AE-reject outcome. (1) StateReport heartbeat-reply now carries the standby's current term (controller.capnp +term field @2, typemap.h state_report_t +term, ha_nop_reply_state_report populates it from ha->term) so the primary learns a higher term on every convergence cycle. (2) New library entry point raft_on_higher_term_observed(s, observed_term) mirrors UpdateTerm (strictly-greater-than only; touches term/voted_for/role; on_persist before on_become_follower for durability ordering) and is called from repl_ack_decode_and_dispatch on both the heartbeat state-report branch and the WAL-NACK branch (ha.c). Primary now steps down instead of retrying AppendEntries against a follower that rejects forever. Validated: 265 controller + 287 lib unit tests pass (zero regressions); 10-round sequential soak PASS (33 jobs, zero loss, all 3 nodes led ctrl-1/ctrl-2/ctrl-3); 10-round chaotic soak PASS (33 jobs, zero committed divergence, zero committed job loss).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1417, bug-1418, bug-851

38
.wolf/bugs/bug-1422.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1422"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "unknown"
severity: "unknown"
timestamp: "2026-07-09T10:14:14.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: [".claude/skills/xce-test-ha/run-ha-soak.sh"]
tags: ["xce-test-ha", "false-positive", "client-failover", "retry-budget-exhaustion", "E_XCE_STANDBY", "test-harness"]
related: ["bug-854", "bug-1421"]
legacy: true
---
# bug-1422: 40-round sequential HA soak FAIL at round 36: verify_no_loss reported '0/111 jobs confirmed…
## Description
40-round sequential HA soak FAIL at round 36: verify_no_loss reported '0/111 jobs confirmed present' right after a fresh election (new primary ctrl-1, term=40) — xce_test's expect-job got E_XCE_STANDBY (10014) from a fresh client session whose own wait-primary had JUST confirmed ctrl-1 as serving 45ms earlier
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
False-positive test-harness failure, NOT real committed-job loss: ctrl-1's own controller log confirms it stayed CTRL_PRIMARY throughout the window and successfully committed the 3 new post-failover jobs (log_append/ae_reply_success/commit_advanced at 10:14:22-23). ha_dispatch_gate's CTRL_PRIMARY branch (ha.c:3451-3531) unconditionally permits OPCode_get (joblist) even while fenced, so ctrl-1 itself could not have returned E_XCE_STANDBY for this query — the reply must have come from a standby (ctrl-2, just killed, or ctrl-3) that the client's failover retry loop landed on. Root cause traced to client_failover.c's xcecli_retry_next/xcecli_retry_decide (client_failover.c:305-319): E_XCE_STANDBY classifies as XCE_FO_RETRY_NOW (client_failover.c:59-60, prompt rotate, no backoff), and the 20-attempt retry budget (3 masters) should reliably reach the real primary — but when the WHOLE budget is exhausted while every recent classification was RETRY_NOW (as opposed to RETRY_BACKOFF), xcecli_retry_decide's budget-exhaustion branch (line 312-318) does NOT override *rc the way it does for RETRY_BACKOFF exhaustion (which maps to the distinct -E_XCE_NOT_READY, line 316-317) — it just returns false with *rc left at whatever the last attempt's raw status was (here, a stray E_XCE_STANDBY from a standby hit during the rapid post-election retry storm). verify_no_loss() (run-ha-soak.sh) then did a single one-shot check with zero retry at the driver level, unlike the job-submit loop just above it in the same file which already retries on this exact 'primary likely just promoted, regaining write-quorum' condition (run-ha-soak.sh:483-486). Only surfaced at round 36 of a 40-round run — narrow enough timing that shorter 6-10 round soaks across many prior sessions never hit it.
## Fix
Added a bounded retry (3 total attempts, 2s settle sleep between) around the WHOLE verify_no_loss() check in run-ha-soak.sh, mirroring the existing submit-retry loop's rationale: a job still missing after a real settle window + 2 retries is a genuine loss, not this client-side retry-storm race. Did not touch client_failover.c (the battle-tested libxce-failover retry/rotation library with many other production consumers) — fixing at the test-driver level is safer and fully local. Re-ran both 40-round soaks after the fix: sequential PASS (123 jobs, zero loss, all 3 nodes led) and chaotic PASS (123 jobs, zero committed divergence, zero committed job loss).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-854, bug-1421

38
.wolf/bugs/bug-1423.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1423"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-07-09T13:58:16.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["ha", "raft", "election", "liveness-deadlock", "vote-request", "dead-peer", "quorum", "permanent-stall"]
related: ["bug-1417", "bug-1418", "bug-1421"]
legacy: true
---
# bug-1423: 30-round sequential HA soak FAIL at round 5: post-failover submit did not get 3 acks after 5 retry…
## Description
30-round sequential HA soak FAIL at round 5: post-failover submit did not get 3 acks after 5 retry attempts (15s+). Root-caused as a genuine permanent election deadlock, not a client-side false positive: ctrl-3 won PreVote and entered CTRL_ELECTING (real election, term=10) with ctrl-1 dead and ctrl-2 alive, then never showed another HA state-machine event for 2+ minutes (until the harness killed the cluster on FAIL) — stuck in CTRL_ELECTING forever. Meanwhile ctrl-2 endlessly retried its own PreVote every ~15-17s, correctly denied every time by ctrl-3 on freshness grounds, but that denial never unstuck ctrl-3's already-in-flight real election.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
raft_cb_send_vote_request (ha.c, sends the real RequestVote RPC to each peer) has two failure branches — endpoint_connect() failing and iomgr_monitor() failing — that are exactly what happens when the RPC target is a dead/unreachable peer (here, ctrl-1, killed at the start of round 5). The endpoint_connect failure branch only called the legacy handle_request_vote_reply() when `c == NULL` (unit-test mode) — in PRODUCTION (c always non-NULL) a connect failure to a dead peer was silently dropped with NO notification to the raft library at all. The iomgr_monitor failure branch called handle_request_vote_reply() unconditionally, including in production — but per vote_reply_dispatch's own design comment, handle_request_vote_reply is the LEGACY/ctx-done-signaling path; 'raft_on_vote_reply remains the SOLE production promotion driver' (vote_reply_dispatch calls ha_record_vote_reply + raft_on_vote_reply, NOT handle_request_vote_reply). Net effect: a candidate's vote request to a dead peer never registers as a 'reply' with the production raft library's raft_state_t.replies_received counter. raft.c's Step 6 fast-path ('all peers have replied, no quorum → become follower and retry', gated on replies_received == n_peers-1) can therefore NEVER trigger whenever any configured peer is down — with quorum=2 in a 3-node cluster and one peer dead, replies_received caps at 1 (only the live peer's real reply, if any) and can never reach n_peers-1=2. Recovery then depends ENTIRELY on the raw per-campaign election_timeout (5-10s, ha_jittered_election_delay) firing via delaymgr — in the observed failure, no election_timeout_fn trace appeared in the log at all for the full 2+ minute window, leaving the candidate permanently wedged.
## Fix
Routed both raft_cb_send_vote_request failure branches (connect failure and monitor failure) through vote_reply_dispatch(ha, ectx, votes_idx, term, vote_granted=0, 0, 0) in production (c != NULL) — the exact same call a genuine vote reply makes (ha_record_vote_reply + raft_on_vote_reply), so a dead/unreachable peer now correctly counts as 'replied, not granted' immediately on connect/monitor failure instead of only via the timeout. Test mode (c == NULL) keeps the legacy handle_request_vote_reply call unit tests depend on. Added a forward declaration of the previously-file-local-only vote_reply_dispatch near the top of ha.c so raft_cb_send_vote_request (defined much earlier in the file) can call it. Verified: full controller_unit_test suite clean (755 PASS / 0 Fail, exit 0) both before and after removing temporary diagnostic logging used during root-cause; re-ran the exact failing scenario — 30-round sequential HA soak PASS (93 jobs, zero loss, all 3 nodes led) and 30-round chaotic HA soak PASS (93 jobs, zero committed divergence, zero committed job loss) — both previously hit this stall reliably (round 5 of the sequential run, 100% reproduction before the fix).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1417, bug-1418, bug-1421

38
.wolf/bugs/bug-1424.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1424"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-09T17:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["ha", "raft", "prevote", "dead-peer", "latency"]
related: ["bug-1423"]
legacy: true
---
# bug-1424: PreVote twin of bug-1423: raft_cb_send_pre_vote_request's connect/monitor failure branches (a…
## Description
PreVote twin of bug-1423: raft_cb_send_pre_vote_request's connect/monitor failure branches (a dead/unreachable peer) never registered a reply with the raft library's PreVote tally, in EITHER test or production mode (worse than the real-vote path, which at least handled test mode).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Same missing-registration pattern as bug-1423, in raft_cb_send_pre_vote_request instead of raft_cb_send_vote_request. Found while auditing for the analogous gap after fixing bug-1423. Lower severity than bug-1423: PreVote has an unconditional round-timeout fallback (ha_prevote_round_timeout_fn, confirmed firing reliably every round across dozens of observed rounds in the bug-1423 investigation logs) that always resolves the round regardless of tally completeness, so this was never a permanent-deadlock class bug like bug-1423 was — just a latency waste: every PreVote round against a down peer burns the full round timeout instead of concluding as soon as the live peer(s) reply.
## Fix
Routed both failure branches through pre_vote_reply_dispatch(ha, c, peer_idx, granted=0) — the same production PreVote-reply path a real reply uses (raft_on_pre_vote_reply) — mirroring bug-1423's fix exactly. Added a forward declaration near the top of ha.c (pre_vote_reply_dispatch is defined much later in the file). Verified: full controller_unit_test suite clean (0 Fail, exit 0) after the change.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1423

60
.wolf/bugs/bug-1425.md Normal file
View file

@ -0,0 +1,60 @@
---
id: "bug-1425"
status: "fixed"
status_updated: "2026-09-24"
status_note: "8a6c339a"
component: "controller"
severity: "hang"
timestamp: "2026-07-09T21:15:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/persist.c or src/controller/commit.c (unrooted)"]
tags: ["ha", "commit-engine", "election", "intermittent", "raft-apply-decoupling-followup", "live-repro-found", "concurrent-connect-trigger", "root-caused", "tdd-red-green"]
related: ["bug-1416"]
fix_commits: ["8a6c339a"]
legacy: true
---
# bug-1425: Intermittent (~2 of 9 attempts) client hang in the synchronous jobsub (COMMIT_MODE_WAIT) path…
## Description
Intermittent (~2 of 9 attempts) client hang in the synchronous jobsub (COMMIT_MODE_WAIT) path shortly after a fresh single-node election.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
NOT ROOT-CAUSED. Flagged during raft-apply-decoupling task 7.4's throughput benchmark work. Reproduced identically on both the pre-fix and post-fix (task 6.1 apply-pump) binaries — confirmed unrelated to that fix. Resembles but is not confirmed identical to the closed bug-1416 boot-race class (persist_recover() replay-verify racing the live apply pump / live-replication apply path).
## Fix
[UPDATE 2026-08-16] Live investigation (single-node deploy via /xce-deploy-ha, controller_unit_test's own fix build): found a RELIABLE reproduction recipe the original ~2/9-sequential-attempts description never had, and narrowed the root cause to a specific mechanism, but did NOT land a fix -- reporting findings honestly per the explicit scope of this investigation pass.
RELIABLE REPRO (not sequential -- CONCURRENT): deploy a fresh single-node cluster, poll the log for CTRL_PRIMARY, then fire 9 xce_test client processes (each: connect + one jobsub) SIMULTANEOUSLY (backgrounded, not sequential) immediately after election. Reproduced in 3 of 4 rounds (1-3 of 9 concurrent clients failing per round, closely matching the original ~2/9 ratio) -- 0/45 reproductions when the SAME 9 attempts were fired sequentially instead (one xce_test process at a time). Concurrency at connect time, not raw timing after election, is the actual trigger.
SYMPTOM, precisely: the failing client's xcecli_jobsubmit() (client.c, wrapped in XCECLI_FAILOVER_CALL) returns rc>=0 (success-shaped, NOT a negative error) after exactly ~10.0-10.1s, with the output jobid=0 (never written) instead of the submitted JobId -- test.c's cmd_jobsub (bug-786's own guard) correctly catches this as 'no clean ack' rather than misreporting success, so no data is actually lost/corrupted -- but the client hangs ~10s before getting a clear answer, matching the bug's own description exactly.
CONFIRMED CLIENT-SIDE, not server-side: cross-referenced the controller's own log (fresh --logfile per round) against the 9 concurrent attempts -- server-side WAL only ever shows 8 job-submit entries committed (lsn sequence has no gaps, no admission-cap rejection WARN, no error of any kind) when 9 concurrent clients attempted -- the failing 9th request never reaches handle_jobsubmit at all. Ruled out the obvious admission-cap suspect: commit.c's COMMIT_MODE_WAIT admission control (cap=n_vcpus-VTASK_LIVENESS_RESERVED=3 on this 4-vcpu deploy) never logged its 'WAIT rejected, admission cap exhausted' WARN in any reproduction -- 8 concurrent WAITs completing well under the cap is plausible given each takes <100ms server-side once received.
THE ~10s NUMBER, precisely sourced: xceapi.c's xce_jobsubmit() (the innermost call XCECLI_FAILOVER_CALL wraps) has a HARDCODED cbuf_deadline_10s() wait on cbuf_timedget waiting for a reply to land in the single-slot ctx->wbuf. Read cbuf_timedget itself (mylock.c) -- confirmed CORRECT, it returns -ETIMEDOUT (negative) on a genuine timeout, which xce_jobsubmit correctly propagates as a negative rc. So a bare cbuf_timedget timeout cannot by itself explain the observed rc>=0-with-jobid=0 shape; something in the retry/rotate/recover layer ABOVE it must be involved.
NARROWED SUSPECT (not confirmed): client_internal.h's own documented CI1 hazard -- 'handle_disconnect may have injected a stale status into the single-slot ctx->wbuf ... xcecli_ctx_recover drains any pending entry with a non-blocking cbuf_tryget so the next attempt's cbuf_get cannot consume a prior-attempt reply' -- but this drain only runs when xce_failover_rotate() succeeds (client_failover.c:130-153, mirrored by xcecli_retry_next per its own doc comment: 'only recover when the rotate succeeded'). On this single-candidate cluster, rotate reconnects to the SAME (only) node via a real xce_ctx_reconnect() -- plausible this usually succeeds (ruling out the simplest 'rotate never succeeds on single-node, drain never runs' theory) but NOT independently verified under the exact concurrent-connect contention this repro needs; a transient reconnect failure specifically during the connection-establishment storm (9 simultaneous TLS handshakes against a listener seconds old) could leave a stale wbuf entry undrained, which a LATER retry's cbuf_timedget could then either (a) consume immediately (fast, not 10s -- doesn't match) or (b) never resolve at all if the drain path interacts badly with an in-flight send, leaving the single wbuf slot permanently empty until the full 10s deadline. Did not instrument (fprintf/gdb) the actual attempt count or wbuf state during a live failing run to confirm this mechanism specifically -- xce_test's own -vvv flag produced no additional client-side retry/rotate trace output (goes through a different, unwired logger in this CLI context), and adding real instrumentation would need a fresh lib rebuild + re-repro cycle not completed in this pass.
NOT YET DONE: instrument xcecli_retry_next/xce_jobsubmit directly (attempt count, wbuf drain outcome, rotate rc, per-attempt elapsed) and re-run the now-reliable concurrent-burst repro to catch the exact mechanism in the act -- this is the recommended next step, using the repro recipe above (no gdb needed; the reproduction is now reliable enough for print-based debugging). Confirmed unrelated to raft-apply-decoupling task 6.1's apply-pump fix (reproduces identically on that spec's pre-fix and post-fix binaries per the original entry) and to this session's bug-1428/1426/1474/1472-remainder fixes (none touch xceapi.c/client.c/client_failover.c's client-side retry path).
[Original entry]
NOT FIXED. Worked around via retry for task 7.4's benchmarking purposes only. Needs a dedicated root-cause investigation with the same live-repro rigor bug-1416 got (gdb-attach via GDB_WRAP seam, or targeted debug logging around the commit-wait barrier right after a fresh single-node election).
[UPDATE 2026-08-16, part 2] FIXED. Root-caused via static code trace (no gdb/instrumentation needed after all): XCECLI_FAILOVER_CALL's idempotent-replay rule (client_failover.c, xce_failover_classify -- 'a duplicate on a re-sent write means the replay guard already applied our write, treat as success') can normalize a retried resend that hits the server's EEXIST/inflight-collision rejection into overall success (rc==0) -- but that rejection reply carries NO payload (handle_jobsubmit's EEXIST/inflight-collision branches reply with RequestReply_data_nodata, job.c:1441-1467), so xce_jobsubmit() (xceapi.c) never populates its *jobid output on the winning (dup) attempt, and never did on the earlier failing attempt either (its raw status was already negative). Net: xcecli_jobsubmit() can return rc>=0 with *jobid left at the caller's initial value (0) -- matching the confirmed symptom exactly (rc>=0, jobid=0, ~10.0-10.1s: one full 10s cbuf_timedget deadline on the failing first attempt, then a fast dup-rejection reply on the resend).
FIX (TDD): added jobsubmit_finalize_jobid(uint64_t* jobid, uint64_t submitted_jobid, int overall_rc) (src/lib/client_internal.h declaration, src/lib/client.c definition), called from xcecli_jobsubmit() right after XCECLI_FAILOVER_CALL returns. Sets *jobid = submit->jobid (the CLIENT-generated id already on the wire -- job ids are client-generated via idgen(); the controller commits/echoes this exact value verbatim on every genuine success reply, bug-786/job.c:1439) whenever overall_rc >= 0, otherwise leaves *jobid untouched. Deliberately placed at the jobsubmit-specific wrapper layer, not the generic classify/decide layer (which is correctly type-erased and has no concept of jobid) -- test.c's cmd_jobsub (bug-786's own gate, comparing returned jobid == locally-generated jobsubmit.jobid) was ALREADY the right verification signal and needed no changes. 3 new pure unit tests in test_libxce_retry.c (test_jobsubmit_finalize_populates_jobid_on_success/leaves_jobid_untouched_on_failure/null_jobid_safe) -- socket-free, following this codebase's established direct-state-simulation test style for I/O-adjacent logic (test_netio_on_connected.c's precedent) rather than mocking a live connection.
LIVE VALIDATION: full ASAN rebuild (0 compiler errors; controller_unit_test 1101 tests, 2 pre-existing unrelated test_config_repl.c failures; libxce test suite 1/1 OK, all 3 new tests PASS). Deployed a fresh single-node cluster and ran the same concurrent-burst repro as the original investigation (fresh election -> 9 simultaneous xce_test jobsub clients) for 5 rounds / 45 total clients: 0 'no clean ack' failures, 45/45 clean acks with jobid matching the submitted id (vs. the original repro's 1-3/9 failures in 3 of 4 rounds pre-fix). Honest caveat: this is probabilistic/timing-dependent (like the original repro), so 45/45 clean is strong but not airtight proof the exact race window is unreachable -- the fix is correct by construction regardless (it closes the *jobid gap for EVERY path that can produce rc>=0, not just the one race that was reproduced), so residual risk is limited to a still-undiscovered SEPARATE mechanism producing the same symptom, not this one going unfixed.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1416

41
.wolf/bugs/bug-1426.md Normal file
View file

@ -0,0 +1,41 @@
---
id: "bug-1426"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-09T21:15:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c (trim_wal_segments", "apply_pump_read_entry cache)"]
tags: ["ha", "persistence", "wal-trim", "apply-pump", "low-priority", "raft-apply-decoupling-followup"]
related: []
legacy: true
---
# bug-1426: trim_wal_segments's retention floor has no awareness of the local apply pump's own lag — a…
## Description
trim_wal_segments's retention floor has no awareness of the local apply pump's own lag — a reachable scenario since apply-lag-behind-commit is raft-apply-decoupling's entire premise.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Identified during raft-apply-decoupling task 7.4's apply-pump segment-read cache fix (round-2 review finding): the new fast-path cache can (POSIX unlink-keeps-fd-alive) silently advance last_applied_lsn past already-deleted data if the pump is lagging when trim_wal_segments fires. Currently bounded by the pre-existing APPLY_DIVERGENT boot fence (deferred loud fencing on next restart, not silent corruption) — a real behavior change from the pre-cache code, not literally risk-free, but not an active defect either. Same family as an already-tracked standby-cursor variant (per task 7.4's own notes).
## Fix
FIXED (TDD RED->GREEN). trim_wal_segments() now caps its trim floor at the local apply pump's own progress (last_applied_lsn), not just remote standby-acked positions: floor = min(repl_trim_lsn's standby/disconnect-guard floor, last_applied_lsn+1). The +1 matters -- repl_trim_lsn's floor treats equality as 'still needed, preserve' (a standby acked through LSN N might still need that segment for retry/resync), but the LOCAL apply cursor's equality case means the opposite: last_applied_lsn==N means this process is DONE with entry N and will never re-read it, so a segment starting exactly at N is safe to delete. Feeding last_applied_lsn+1 into the same shared 'seg_start_lsn < floor' comparison gets correct delete-on-equal semantics for this term without a second, differently-shaped check. This closes the real gap: apply_pump_read_entry()'s fast-path cache holds an open fd into whatever segment it's currently reading; POSIX unlink-keeps-fd-alive meant a lagging pump could keep reading via that stale fd even after trim_wal_segments deleted the segment out from under it (no immediate corruption, since the fd's data was still valid) -- but a FRESH read of the same range (a restart's persist_recover() replay, or the pump's own cache falling back to its slow path after an unrelated miss) would find the segment already gone, producing a genuine APPLY_DIVERGENT-class gap between what the live process quietly caught up to and what a fresh read of disk state can see. Regression test (test_e2e_checkpoint.c's test_trim_guard_preserves_unapplied_local_segments): no standbys registered (so the pre-fix floor alone would allow trimming everything below the active segment) but last_applied_lsn deliberately lags behind one segment -- proves the already-applied segment IS trimmed (no unnecessary retention) while the not-yet-applied one is preserved (the actual guard). Also had to advance last_applied_lsn in the 3 pre-existing trim tests in the same file (test_checkpoint_wal_trim_after_checkpoint, test_trim_guard_preserves_standby_segments, test_trim_without_standbys_proceeds_normally), which call persist_wal_async directly (bypassing the whole apply path) and so never advance last_applied_lsn on their own -- without the added stamp, this fix would have correctly-but-unexpectedly blocked ALL trimming in those fixtures, since last_applied_lsn defaults to 0. Full controller_unit_test suite: 1099 tests, 2 pre-existing unrelated failures (bug-1561), no regressions.
[Original entry]
NOT FIXED — deliberately deferred. task 7.4's own scope explicitly excluded redesigning trim_wal_segments's retention-floor awareness of apply-pump lag; documented via an accurate code comment instead. A future fix would give trim_wal_segments a floor input derived from the local apply cursor, not just standby-acked LSNs.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

41
.wolf/bugs/bug-1427.md Normal file
View file

@ -0,0 +1,41 @@
---
id: "bug-1427"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-09T21:15:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c (persist_verify_apply_correctness", "count_jobmgr_entries)"]
tags: ["ha", "persistence", "apply-divergent", "worker", "low-priority", "raft-apply-decoupling-followup"]
related: []
legacy: true
---
# bug-1427: APPLY_DIVERGENT's entity-count correctness check (persist_verify_apply_correctness) covers job…
## Description
APPLY_DIVERGENT's entity-count correctness check (persist_verify_apply_correctness) covers job entities only — worker entities can diverge (worker_from_capnp()/restore_worker() genuinely creates a new worker on a WAL_WORKER_CTRL/HOOK_DETAIL_WORKER_JOIN entry when absent) but this is currently invisible to the check.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Disclosed, accepted scope gap from raft-apply-decoupling task 5.1's own design: the entity-count check was scoped to jobs only (queues correctly excluded — config-derived, WAL_QUEUE_CTRL never creates a new one, verified by two review rounds). Workers were never brought in scope. Not a regression — a known, deliberate boundary, just not yet extended.
## Fix
FIXED (TDD RED->GREEN). Extended persist_verify_apply_correctness() with a worker-count check mirroring the existing job-count check exactly: added count_workermgr_entries() (persist.c, mirrors count_jobmgr_entries()), a new expected_new_worker_count field on apply_expectation_t (persist.h), and a new workermgr_count_before_replay parameter threaded through persist_verify_apply_correctness()/persist_recover(). The harder part: apply_committed_entry()'s WAL_WORKER_CTRL case previously always returned 0 regardless of outcome (JOIN-creates-new, UPDATE-applies-to-existing, and idempotent-noop were all indistinguishable to callers), so replay_wal() had no signal to tally new worker joins from. Fixed by making worker_from_capnp() (worker.c) return 1 specifically for a genuine new-worker creation (discarding restore_worker()'s raw idx, which can legitimately be 0 and would have collided with the update-applied case) and propagating that rc through apply_committed_entry() instead of the hardcoded 0 -- verified no caller of either function depended on the exact prior return value (only sign and -EEXIST were ever checked). replay_wal() tallies apply_rc==1 for WAL_WORKER_CTRL exactly like it already does for WAL_JOB_SUBMIT. New regression tests: test_verify_apply_correctness_detects_worker_count_mismatch (test_apply_correctness_verification.c, mirrors the existing job-count-mismatch test) proves a worker-count divergence is now caught as APPLY_DIVERGENT -- RED before this fix (no worker check existed at all), GREEN after. Also updated test_apply_committed_entry.c's test_ace_worker_ctrl_join_restores_and_notifies, which asserted rc==0 for a genuine JOIN creation under the old (now-corrected) contract -- now asserts rc==1 with an explanatory message. Full controller_unit_test suite: 1097 tests, 2 pre-existing unrelated failures (bug-1561), no regressions.
[Original entry]
NOT FIXED. Extend persist_verify_apply_correctness() with a worker-count check mirroring count_jobmgr_entries() if/when a worker-creation divergence needs to be closed (task 5.1's own forward note).
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

41
.wolf/bugs/bug-1428.md Normal file
View file

@ -0,0 +1,41 @@
---
id: "bug-1428"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "security"
timestamp: "2026-07-09T21:15:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c (persist_register_pending_apply)"]
tags: ["ha", "persistence", "apply-pump", "toctou", "unconfirmed", "raft-apply-decoupling-followup"]
related: ["bug-1415"]
legacy: true
---
# bug-1428: Narrow TOCTOU residual in persist_register_pending_apply() (task 6.2, part of bug-1415's fix)…
## Description
Narrow TOCTOU residual in persist_register_pending_apply() (task 6.2, part of bug-1415's fix) — not reliably exercised by any current test.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Flagged during raft-apply-decoupling task 6.2 as a non-blocking fast-follow while closing bug-1415. Specific race window not fully documented in tasks.md's own notes; needs a fresh look at the function to characterize exactly what window remains and whether it's reachable in production timing.
## Fix
FIXED (characterized then TDD RED->GREEN, per the entry's own 3-step plan). Characterized the exact window: persist_register_pending_apply()'s self-heal check (an UNLOCKED read of last_applied_lsn) ran BEFORE acquiring pending_applies_lock, separate from the locked insert that followed. Window: this thread reads last_applied_lsn (sees 'not yet applied') -> BEFORE it locks and inserts, the apply pump's thread advances last_applied_lsn past this lsn AND completes its own persist_take_pending_apply(lsn) call (finding nothing, since this thread hasn't inserted yet) -> this thread then inserts a node the pump will never revisit -- permanently orphaned, identical failure mode to bug-1415 (a hang, on_applied never fires) via a different, narrower trigger window. Production reachability: yes -- persist_register_pending_apply() is called from a different thread/vtask than the apply pump's walk (commit.c's continuation dispatch vs persist_apply_pump_run, per the function's own pre-existing doc comment), with no synchronization between them before this fix beyond the already-locked insert. Fix: moved the self-heal check inside pending_applies_lock, so the whole check-then-(insert-or-self-heal) sequence is one atomic critical section against persist_take_pending_apply's own locked scan-and-remove -- either this thread's check now runs before the pump's take (sees 'not yet applied', inserts, and the pump's later take, now forced to wait on the same lock, finds the freshly-inserted node) or after (sees 'already applied', self-heals instead of inserting, and the pump's take correctly finds nothing) -- never the interleaved gap that orphaned it. Regression test (test_apply_pump.c's test_register_pending_apply_race_against_concurrent_apply_self_heals): deterministic reproduction via controlled state (a main thread holding pending_applies_lock forces the register call to observe last_applied_lsn only after the lock's own update, removing dependence on real-time race timing) -- RED under the pre-fix code (on_applied never fires, node orphaned on pending_applies), GREEN under the fix (self-heals correctly, no orphan). Full controller_unit_test suite: 1098 tests, 2 pre-existing unrelated failures (bug-1561), no regressions.
[Original entry]
NOT FIXED — explicitly deferred as a fast-follow, not a blocker for task 6.2 or the spec's closure. Needs: (1) characterize the exact TOCTOU window, (2) determine production reachability, (3) add a regression test that exercises it before fixing.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1415

54
.wolf/bugs/bug-1429.md Normal file
View file

@ -0,0 +1,54 @@
---
id: "bug-1429"
status: "fixed"
status_updated: "2026-09-25"
status_note: "b5d59253"
component: "controller"
severity: "crash"
timestamp: "2026-07-10"
last_seen: "2026-09-24T10:51:31Z"
occurrences: 5
files: ["src/controller/testsuite/test_commit_barrier_rollout.c", "test_demotion_convergence.c", "test_convergence_verification.c", "test_egress_filter.c", "test_apply_pump.c", "test_apply_pump_integration.c", "test_job_array_fence.c", "test_ha_role_lifecycle.c", "test_job_array_replay_fence.c", "test_commit_durability.c", "test_commit_engine.c", "test_p2_integration.c (test-fixture teardown ordering", "21 sites across 12 files); src/controller/commit.c (commit_engine_fini", "unchanged -- confirms the documented contract); src/controller/persist.c (fsync_worker", "unchanged -- confirms the race window)"]
tags: ["FIXED", "VTASK_LIVE_BLOCK", "commit_engine_fini", "config_repl.c", "controller-tests", "delaymgr", "fsync_worker", "gdb-confirmed", "ha_convergence.c", "hang", "live-repro", "multi-location", "persist.c", "persist_stop", "root-caused", "teardown-ordering", "test-fixture-bug", "test_commit_barrier_reply", "timeout", "toctou-race", "vtask"]
related: ["bug-1452", "bug-1687", "bug-924"]
fix_commits: ["b5d59253"]
legacy: true
gitea_issue: 264
gitea_synced_status: "fixed"
---
# bug-1429: controller_unit_test suite TIMEOUT (300s, killed by SIGTERM) during /xce-build controller run…
## Description
controller_unit_test suite TIMEOUT (300s, killed by SIGTERM) during /xce-build controller run immediately after uncommitted edits to persist.c, ha_convergence.c, config_repl.c (delaymgr-inline blocking I/O moved to VTASK_LIVE_BLOCK vtasks with inline-fallback-on-failure). Compile succeeded with zero errors (only pre-existing warnings). meson test summary: Ok:0 Fail:0 Timeout:1. Last test output before the hang was test_demotion_replay_yields_committed_only (test_commit_barrier_reply.c) printing overlayfs-SKIP, followed by a burst of HA FSM / persist_apply_pump_run log lines ending in "HA quorum lost: connected=0 required=1", then total silence for the remaining ~294s until SIGTERM. No ASAN/crash report -- looks like a genuine hang/deadlock, not a crash.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ROOT-CAUSED 2026-08-30, LIVE-REPRODUCED with a full gdb thread backtrace (the first time in this bug's history any session captured a live hang instead of just reporting the SIGTERM/timeout). The original delaymgr/VTASK_LIVE_BLOCK hypothesis carried by this bug since 2026-07-10 was WRONG -- a static read of every vtask_enq_classified(..., VTASK_LIVE_BLOCK) call site in ha.c/persist.c/config_repl.c/commit.c found every one of them NULL-checks its pool pointer (or the trampoline is gated behind `c->delaymgr.running`, which test fixtures never set) and falls back inline on failure -- there is no path where a controller_unit_test fixture leaves a non-NULL-but-never-started vtaskmgr_t that a VTASK_LIVE_BLOCK task could be silently swallowed into. That entire subsystem is a red herring for this bug.
The REAL root cause is the same TOCTOU test-teardown race already root-caused once before, for a DIFFERENT symptom, in bug-1452 (2026-07-19: an ASAN SEGV in test_persist_truncate.c) and partially re-applied once more in bug-1687 (2026-08-25: cbrp_teardown in test_commit_barrier_reply.c). Every affected test fixture calls `commit_engine_fini(&ce)` BEFORE `persist_stop(&persist)` in its teardown helper -- the exact opposite of the contract documented in commit_engine_fini's own comment (commit.c: 'Callers are expected to have stopped the flusher (persist_stop) first'). Because persist_stop had not yet run, the persist_t's live fsync_worker background thread (persist.c:800, started by persist_start) can still be mid-flight calling `p->on_durable_advance(p->durable_cb_ctx)` -- i.e. commit_on_durable_advance(ce) -- with NO lock held (persist.c:866-868), concurrently with commit_engine_fini destroying ce->lock/ce->cond (commit.c:476-477) and freeing ce->continuations. commit_engine_fini's own persist_set_durable_cb(ce->p, NULL, NULL) unhook is 'defensive and cheap' per its own comment but is NOT sufficient on its own: the flusher thread can have already read the OLD non-NULL callback pointer a few instructions before the unhook takes effect. Depending on exact thread interleaving this TOCTOU race manifests as either bug-1452's SEGV (a NULL/freed dereference inside commit_advance_locked) OR -- newly confirmed this session -- a hang: pthread_cond_destroy(&ce->cond) blocking forever because the racing fsync_worker thread is concurrently inside pthread_cond_broadcast(&ce->cond) / holding ce->lock from commit_on_durable_advance, which glibc's cond-destroy implementation cannot safely proceed past. This explains every symptom previously logged against this bug: the varying hang location across sessions (different tests hit their OWN teardown's copy of this same race, at whatever moment the fsync_worker thread happened to be flushing), the total silence after the hang (both racing threads are genuinely blocked in futex/pthread_cond primitives, not spinning), zero CPU time accruing, and "both threads parked in futex_do_wait" from the 2026-08-23 sighting.
LIVE REPRO (this session): `meson test -C src/xce.build.bug1429 --suite controller --verbose` hung 40s into the run with 3 threads (main + 2x fsync_worker) all in futex_do_wait, 0 CPU time advancing over a 3s recheck. gdb -p <pid> -batch -ex 'thread apply all bt full' (attached BEFORE the 300s SIGTERM, while the process was still alive) showed: Thread 1 (main) stuck in pthread_cond_destroy() called from commit_engine_fini (commit.c:476) <- jap_teardown() (test_commit_barrier_rollout.c:1877) <- test_job_array_submit_payload_size_bounded_at_max_array_idx (test_commit_barrier_rollout.c:2294) <- UnityDefaultTestRun <- test_commit_barrier_rollout() <- main(). Threads 2 and 3 both stuck in pthread_cond_timedwait at fsync_worker (persist.c:827), BOTH with `p = &g_jap` (the same global persist_t) -- confirming jap_teardown's exact commit_engine_fini-before-persist_stop ordering bug (test_commit_barrier_rollout.c:1877-1879, at the time of capture) as the live mechanism. This is the 4th distinct hang location logged against this bug (previously: test_demotion_replay_yields_committed_only, the test_configack_* group, test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs) -- all four are different call sites of the identical teardown-ordering bug, which is why the location never repeated and why every prior static-inference pass aimed at the vtaskmgr/delaymgr subsystem instead never found it.
## Fix
FIXED 2026-08-30. Test-fixture-only fix -- no production code changed (commit.c and persist.c are exactly as documented; the bug is every affected fixture violating commit_engine_fini's own documented calling contract). Inserted an explicit `persist_stop(&persist);` call immediately before every confirmed-vulnerable `commit_engine_fini(&ce);` call (mirroring bug-1452/bug-1687's established fix pattern exactly), leaving each function's pre-existing later persist_stop()/teardown-helper call in place as a safe, idempotent no-op (persist_stop's own fsync_thread_running and wal_fd>=0 guards make the second call a clean skip). Where a fixture also tears down a real multi-vcpu vtaskmgr_t pool (test_apply_pump.c's tpr_teardown, test_apply_pump_integration.c's tpi_teardown), vtaskmgr_fini() was left in its original position AFTER commit_engine_fini() (a separate, unrelated invariant: no new vtask may be dispatched onto the pool mid-teardown) -- only persist_stop()'s position moved earlier.
21 sites fixed across 12 files, all confirmed vulnerable by verifying (a) the fixture's persist_t is genuinely persist_start()'d (has a live fsync_worker thread) and (b) the teardown called commit_engine_fini() before persist_stop(): test_commit_barrier_rollout.c (3 sites -- g_fce/g_fp inline teardown, cb5_teardown, jap_teardown -- the last being the exact live-reproduced hang site), test_demotion_convergence.c (dc_teardown -- this EXACT site was flagged but explicitly left unfixed by bug-1452's own remediation note as a known follow-up, never actioned until now), test_convergence_verification.c (cv_teardown), test_egress_filter.c (ef_teardown), test_apply_pump.c (2 sites -- tap_teardown, tpr_teardown), test_apply_pump_integration.c (tpi_teardown), test_job_array_fence.c (tf_teardown), test_ha_role_lifecycle.c (lifecycle_cluster_teardown), test_job_array_replay_fence.c (1 site of its 4 persist_start'd fixtures -- the other 3 never wire a commit engine), test_commit_durability.c (3 sites, including one with a comment that incorrectly claimed persist_set_durable_cb's unhook alone was sufficient), test_commit_engine.c (3 sites, all sharing sync_persist_teardown), test_p2_integration.c (3 sites, 2 sharing var `p` + 1 using `g_p9`). Confirmed SAFE and left unchanged: barrier_teardown (test_commit_barrier_rollout.c), engine_teardown (test_commit_engine.c), and the fixtures in test_repl_majority.c/test_commit_admission_pool.c/test_coldstart_lsn0.c/test_commit_resolve.c -- all use a 'stub' persist_t that is never actually persist_start()'d (no live fsync_worker thread), confirmed by grep for persist_start against each fixture's global before touching it.
KNOWN REMAINING EXPOSURE (not fixed this session, explicit follow-up for a future pass, same as bug-1452 left one for this session): test_config_repl.c has ~23 call sites following the identical inline `commit_engine_fini(&c->commit); t42_teardown(c);` pattern (t42_teardown, line 629, confirmed calls persist_stop THEN persist_fini -- so every one of these is the same bug) -- not fixed here because the file is large (~7000 lines) with some sites (e.g. the c_control/c_raced pair around line 2911-2912) not yet individually verified for uniformity, and a bulk edit without that verification was judged too risky for this session's budget. test_term_safety_adapter.c has persist_start'd fixtures (lines 59, 245, 2382) and 5 commit_engine_fini call sites (676, 930, 1776, 1950, 2338) not individually checked. Neither file's exposure has been observed to hang or crash yet (this bug's own history shows the race is intermittent and timing-dependent -- absence of a prior sighting there is not evidence of safety). Recommended next step: a dedicated /xce-audit-invariants or grep-driven sweep of these two files using the exact same verification method used here (confirm persist_start presence, confirm fini-before-stop order, insert persist_stop() immediately before commit_engine_fini()).
VERIFICATION: full `meson test -C xce.build.bug1429 --suite controller` run 3 times back-to-back after the settled fix (no edits during any of these three runs) -- 39.01s / 38.86s / 38.91s, Ok:6 Fail:0 every time, zero hangs, zero timeouts (previously: TIMEOUT 300.08s-300.30s on effectively every attempt this session before the fix, reproduced live on the very first try). No regression test added (per the task's own documented exception for a suite-level timing-dependent teardown race): the race requires a live fsync_worker thread mid-flush at the exact moment of teardown, which is not reliably triggerable in a bounded, fast unit test without reintroducing the same flakiness this fix removes; the 3x full-suite re-run is the verification method this project's own bug-1452 precedent already established for this exact bug class.
UPDATE (2026-09-24): 2026-09-24 ROOT CAUSE of the residual 300s TIMEOUT (live-captured with gdb under a 6-way parallel stress harness, unshare -rn per instance): any failing TEST_ASSERT longjmps out and SKIPS the fixture teardown; the fixture's commit engine sweep thread (and persist fsync_worker) stay asleep on the static engine's ce->cond / p->fsync_cond; the next test re-inits the same static struct underneath them (pthread_cond_init on a condvar with a live waiter), corrupting the waiter count; a later teardown then blocks forever in pthread_cond_destroy (commit_engine_fini / persist_fini) or pthread_join (cb5_teardown) -- or, for a stack-local engine, the sweep thread reads a dead frame (ASAN stack-use-after-return in commit_sweep_worker, test_persist_truncate.c). The failing assertion itself was never seen because Unity's stdout was fully buffered and lost on SIGTERM/abort. Triggering flaky assertions found once visible: test_job_array_submit_one_entry_one_continuation_across_sizes (the test in flight when the 11:34 meson run hung; continuation resolved on the fsync_worker thread, test read a plain int immediately), test_pump_mixed_facts_and_commands_catchup_original_order (reply count read once before trailing continuations), test_persist_flush_fires_durable_advance_callback (fsync_worker credited a concurrent persist_flush advance to itself -> double fire). FIX (uncommitted): (1) controller_test.c line-buffers stdout; (2) XCE_TESTING registries: tearDown() calls persist_test_reap_unstopped() then commit_test_reap_unfinalized_engines(), stopping/joining whatever a failed test leaked (clean full run reaps 0; injected failure now = 1 FAIL in 2s instead of a hang); (3) persist.c flusher_flush_once reports only its own advance (fsync_worker no longer double-fires; regression test test_flusher_pass_does_not_claim_concurrent_flush_advance); (4) the two flaky tests wait (bounded) for cross-thread resolution.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1452, bug-1687, bug-924
- Gitea: #264

38
.wolf/bugs/bug-1430.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1430"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-07-10T07:12:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c"]
tags: ["asan", "use-after-free", "prevote", "delaymgr", "chaotic-soak", "raft"]
related: ["bug-1423", "bug-1424"]
legacy: true
---
# bug-1430: AddressSanitizer: heap-use-after-free on address ... WRITE of size 4 ... #0…
## Description
AddressSanitizer: heap-use-after-free on address ... WRITE of size 4 ... #0 ha_prevote_cancel_round_timeout ../controller/ha.c:6712 #1 ha_start_prevote_async ../controller/ha.c:6889 #2 ha_failover_stagger_fn ../controller/ha.c:7259 -- crashed 2 of 3 nodes during a live chaotic HA soak (10-round overlapping-kill mode)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_prevote_round_timeout_fn (the delaymgr task body for a PreVote round's timeout) only cleared the static s_prevote_round_ctx pointer inside its 'round still in progress' branch (ha->raft.pre_vote_in_progress true). When a round instead got decided via a reply arriving through raft_on_pre_vote_reply (which clears pre_vote_in_progress without touching s_prevote_round_ctx) and the round-timeout task fired afterward anyway (a benign, expected race), the function fell straight through to free(ctx) with s_prevote_round_ctx still pointing at that about-to-be-freed memory. The next PreVote round's ha_start_prevote_async defensively calls ha_prevote_cancel_round_timeout() before arming its own timer, reading the dangling pointer and writing to old->cancelled -- UAF. Same call stack (single delaymgr thread T8) on both crashing nodes, all-100%-reproducible under chaotic-mode's rapid overlapping restarts.
## Fix
ha_prevote_round_timeout_fn now does a CAS (atomic_compare_exchange_strong_explicit matching s_prevote_round_ctx against this specific ctx, clearing to NULL) UNCONDITIONALLY as the first action, before either branch -- so the static pointer is never left dangling regardless of which path is taken. Brings it in line with the already-correct pattern used by the sibling function raft_election_timeout_fn (same file), which unconditionally clears its own stored ctx pointer before checking cancelled.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1423, bug-1424

38
.wolf/bugs/bug-1431.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1431"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "data-loss"
timestamp: "2026-07-10T10:02:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/controller/ha.c", "src/controller/persist.c", "src/include/persist.h"]
tags: ["raft", "promotion", "committed_lsn", "chaotic-soak", "apply-decoupling", "durability"]
related: ["bug-1432"]
legacy: true
---
# bug-1431: chaotic HA soak: committed_job_loss=5-8 across 10 rounds (VERDICT result=FAIL) after…
## Description
chaotic HA soak: committed_job_loss=5-8 across 10 rounds (VERDICT result=FAIL) after raft-apply-decoupling task 16 gated standby apply strictly on commit; committed_divergence stayed 0 and all jobs eventually reappeared at final settlement -- a visibility-latency gap, not true data loss
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Two stacked gaps in raft_cb_become_leader promotion, both masked pre-task-16 by eager apply-on-WAL-receipt: (1) persist_truncate_wal_above only sets p->committed_lsn on branches of truncate_wal_above_locked that perform a REAL truncation; its common no-op early-return (leader own log already covers safe_lsn) left committed_lsn stuck at its pre-election value, so newly-computed safe_committed_lsn was never applied to the actual commit watermark. (2) Even after fixing (1) with a durable_lsn-capped store, a newly-promoted leader whose own fsync had not yet caught up to its own WAL tip (group-commit batches fsync on a timer) stayed capped below safe_lsn indefinitely -- nothing re-drove the promotion advance once the flusher eventually synced, since fsync_worker on_durable_advance only re-evaluates the commit engines PENDING per-entry continuations, and a standby has none for replicated entries.
## Fix
New function persist_promotion_advance_committed(p, safe_lsn) (persist.c, declared in persist.h) called from raft_cb_become_leader right after persist_truncate_wal_above: unconditionally advances p->committed_lsn toward safe_lsn (monotonic), forcing a synchronous flusher_flush_once(p) first if durable_lsn < safe_lsn (safe/bounded -- by Raft Leader Completeness the new leaders own WAL tip is always >= safe_lsn, so the bytes are already on disk and only need fdatasync). Reduced but did not fully close the soak failures alone -- see bug-1432 for the second half of the fix.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1432

38
.wolf/bugs/bug-1432.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1432"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-10T10:15:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c"]
tags: ["raft", "heartbeat", "promotion", "delaymgr", "chaotic-soak", "latency"]
related: ["bug-1431"]
legacy: true
---
# bug-1432: chaotic HA soak still FAIL (committed_job_loss=8) after bug-1431 force-flush fix; jobs above the…
## Description
chaotic HA soak still FAIL (committed_job_loss=8) after bug-1431 force-flush fix; jobs above the conservative 2-voter safe_committed_lsn floor stayed invisible via expect-job for ~3.1s post-promotion, matching the primary heartbeat interval (3000ms) almost exactly
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_start_primary_heartbeat armed the periodic heartbeat via delay_task_ms(interval_ms, -1, primary_heartbeat_fn, c), whose FIRST invocation fires only after interval_ms elapses (standard periodic-timer semantics) -- despite the functions own long-standing comment claiming to start the heartbeat immediately. A freshly-promoted leader therefore waited a full heartbeat interval before ever sending a NOP to a reconnecting/catching-up standby, so it could not learn that standbys true applied_lsn (via the heartbeat-reply stateReport, ha_process_standby_state) for up to 3s. That knowledge gates when catch-up WAL streaming starts and, in turn, the per-entry majority-ack that advances committed_lsn past the elections conservative safe_committed_lsn floor (jobs above that floor were legitimately majority-committed via the OLD primary before it died, per Leader Completeness, but the new leader cannot expose them without fresh quorum corroboration -- a correct Raft Figure-8 safety requirement, not a bug in itself).
## Fix
ha_start_primary_heartbeat now calls primary_heartbeat_fn(c) synchronously once, immediately, before arming the periodic delay_task_ms timer -- matching the functions own documented intent. primary_heartbeat_fn only performs non-blocking async sends (xce_ha_heartbeat_req), so this is safe inline with no thread-role change. Single fix point covers all 3 promotion call sites (raft_cb_become_leader, ha_promote_to_primary, and the third raft-driven-decision-loop site) since all route through ha_start_primary_heartbeat. Closed the gap: 10-round chaotic soak went from committed_job_loss=8 to committed_job_loss=0, committed_divergence=0, result=PASS; 10-round sequential soak + leadership-coverage also PASS (36 jobs, zero loss).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1431

39
.wolf/bugs/bug-1433.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1433"
status: "fixed"
status_updated: "2026-09-27"
status_note: "255b0faa: peer-control traffic while CTRL_ELECTING classified VTASK_LIVE_BLOCK (become-leader tail preads)"
component: "controller"
severity: "unknown"
timestamp: "2026-07-10T10:59:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c", "src/controller/ha.c"]
tags: ["raft", "promotion", "vtask", "thread-role", "liveness", "audit-finding", "bug-1198-class"]
related: ["bug-1431", "bug-1432", "bug-1198"]
fix_commits: ["255b0faa"]
legacy: true
---
# bug-1433: controller-invariant-audit (2026-07-10, /xce-audit-invariants) async-I/O-thread-role dimension…
## Description
controller-invariant-audit (2026-07-10, /xce-audit-invariants) async-I/O-thread-role dimension: bug-1431 round-2 force-flush fix (persist_promotion_advance_committed calling flusher_flush_once, a blocking fdatasync) is reachable via the real multi-node winning-vote-reply dispatch path on a VTASK_LIVENESS-classified vtask worker -- a task class the design documents as never admission-gated / never expected to block. This is the same bug class as bug-1198 (HA_REPL_SEND_SNAPSHOT persist_checkpoint on VTASK_LIVENESS), which was previously closed by offloading the blocking work off that worker class entirely.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
raft_cb_become_leader (the .on_become_leader raft-library callback) is invoked synchronously from within peer_ctrl_msg_handler -> peer_ctrl_dispatch_to_pool -> peer_ctrl_classify (ha.c ~5935-5940), which HARDCODES VTASK_LIVENESS for every inbound peer-control packet including vote replies -- the doc comment above it explicitly states "everything this handler dispatches is liveness work today" and was written BEFORE this session added a blocking fdatasync call reachable from that same path. ha_dispatch_election_start/raft_election_start_vtask_entry (the VTASK_LIVE_BLOCK dispatch used for the RDHUP-failover and single-node self-grant election-START triggers) does NOT cover the separate, later "win the election / become leader" event, which fires on vote-REPLY arrival via a completely different dispatch chain.
## Fix
Removed the flusher_flush_once force-flush from persist_promotion_advance_committed entirely -- round-3 (durable-capped only, no blocking calls) plus the already-shipped bug-1432 immediate-first-heartbeat fix together close the original visibility-latency gap non-blockingly (confirmed by two live soaks: 6-round chaotic committed_job_loss=0/committed_divergence=0 PASS, 6-round sequential+leadership-coverage PASS). TRACKED RESIDUAL (not fixed this session): persist_apply_pump_run_sync (added earlier this session for the bug-1043-family apply-lag fix, also called from raft_cb_become_leader on the same VTASK_LIVENESS path) does bounded pread() of recently-written WAL segments -- lower practical risk than fdatasync (almost always page-cache-hot) but still, strictly, blocking I/O on a never-blocks task class. Removing it is NOT safe (reopens bug-1043-family job loss). Proper fix needs either reclassifying the winning-vote-reply case in peer_ctrl_classify or dispatching promotion tail onto a fresh VTASK_LIVE_BLOCK vtask -- both larger changes deferred to a dedicated follow-up with its own soak-validation pass. Comments at both call sites corrected to stop falsely claiming VTASK_LIVE_BLOCK safety.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1431, bug-1432, bug-1198

38
.wolf/bugs/bug-1434.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1434"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-10T10:56:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c", "src/controller/ha.c"]
tags: ["raft", "committed_lsn", "race-condition", "monotonicity", "audit-finding", "CAS"]
related: ["bug-1431"]
legacy: true
---
# bug-1434: controller-invariant-audit (2026-07-10, /xce-audit-invariants) Raft TLA+ checklist dimension…
## Description
controller-invariant-audit (2026-07-10, /xce-audit-invariants) Raft TLA+ checklist dimension: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply can race on persist_t.committed_lsn during the promotion window before role flips to CTRL_PRIMARY, via an unlocked load-then-conditional-store pattern in both functions -- a losing racer could clobber a larger value with a smaller one, regressing committed_lsn (violates the documented commitIndex-monotonic invariant).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
raft_cb_become_leader calls persist_promotion_advance_committed BEFORE setting ha->role = CTRL_PRIMARY (the role store sits ~170 lines later in the function). During that window this node is still nominally non-primary, so ha_apply_wal_catchup_entry (still processing an in-flight WAL catch-up entry on another thread, e.g. if this node was mid-catch-up when it won the election) can legitimately call ha_standby_advance_committed_and_apply concurrently with the promotion path own committed_lsn advance. Both did atomic_load(committed_lsn); if(target<=cur) return; atomic_store(committed_lsn, target) with no lock and no CAS, a classic lost-update race under concurrent writers.
## Fix
Added persist_committed_lsn_advance_atomic(persist_t*, uint64_t target) (persist.c, declared in persist.h) -- a CAS retry loop (atomic_compare_exchange_weak_explicit) that is safe against any number of concurrent callers with no lock required, and can never regress committed_lsn regardless of interleaving. Both persist_promotion_advance_committed and ha_standby_advance_committed_and_apply now route through this shared helper. Validated: build clean (19/19 unit test suites pass), 6-round chaotic soak PASS (0 loss, 0 divergence), 6-round sequential+leadership-coverage soak PASS.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1431

38
.wolf/bugs/bug-1435.md Normal file
View file

File diff suppressed because one or more lines are too long

38
.wolf/bugs/bug-1436.md Normal file
View file

File diff suppressed because one or more lines are too long

38
.wolf/bugs/bug-1437.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1437"
status: "open"
status_updated: "2026-09-24"
status_note: "LSan leaks in test_jobj_hardening (makeInstance 240B + 7B strdup), no fix/suppression"
component: "lib"
severity: "crash"
timestamp: "2026-07-13T00:55:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/lib/testsuite/test_jobj_hardening.c"]
tags: ["jobj", "test_jobj_hardening", "leak", "asan", "lsan", "datakiller", "dereference", "hard_build_accounts", "test_job_destroy_does_not_leak_accounts", "unverified-fix", "kvmgr", "job-array-single-wal-record", "mechanical-validation", "reconfirmed"]
related: []
legacy: true
---
# bug-1437: Verification run of lib build + xcelib_test --run test_jobj_hardening…
## Description
Verification run of lib build + xcelib_test --run test_jobj_hardening (ASAN_OPTIONS=detect_leaks=1) after user changed 4x .datakiller=destroy -> .datakiller=dereference in src/lib/testsuite/test_jobj_hardening.c (to fix a leak in the kvmgr-of-job_t test fixture, since job_t.destroy is NULL in RTTI typeinfo and teardown only runs via job_dereference): Unity reports 5/5 PASS, 0 Failures, 0 Ignored, but LeakSanitizer STILL reports two leaks totaling 247 bytes and the process exits 1 (no halt_on_error/abort_on_error set, so no SIGABRT with plain detect_leaks=1 -- differs from the meson full-suite run which sets abort_on_error=1 and gets killed by signal 6).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
TWO leaks detected, only one already known. (1) Direct leak of 240 bytes via calloc -> makeInstance -> makeJob_ (../lib/jobj.c:768) -> makeJob (../lib/jobj.c:789) -> test_job_init_propagates_makeJobsub_failure (test_jobj_hardening.c:133) -- this IS the already-diagnosed, out-of-repo makeInstance-does-not-free-on-negative-init-rc characteristic (vendored rtti library), unrelated to the datakiller fix. (2) Direct leak of 7 bytes via strdup -> hard_build_accounts (test_jobj_hardening.c:269) -> test_job_destroy_does_not_leak_accounts (test_jobj_hardening.c:328) -- this is a SEPARATE, NEW leak signature not previously logged, and it is attributed to the exact test (test_job_destroy_does_not_leak_accounts) that the datakiller=dereference fix was meant to make leak-free. NOT YET DIAGNOSED WHY: could be an unrelated small leak in the hard_build_accounts helper itself (e.g. a strdup()d account string never freed regardless of datakiller correctness), or could indicate the datakiller fix is incomplete. Not investigated further per explicit user instruction to build/run/report only, no fixing.
## Fix
NOT FIXED IN THIS SESSION -- user explicitly instructed build+run+report only, no code changes. Logging the raw finding so the next session does not have to rediscover it: the accounts-leak test still leaks (7 bytes, strdup, hard_build_accounts) even after the .datakiller=dereference change, though it is a much smaller leak than the pre-existing 240-byte makeInstance one and does not cause any Unity test FAILURE (LSAN leak detection is a separate pass/fail axis from Unity PASS/FAIL in this harness). RECURRENCE 2026-07-16 (2nd occurrence, mechanical-validation pass for job-array-single-wal-record spec, dimension A/full-suite reconfirmation): full lib test binary run (ASan) again reports 638 PASS / 0 FAIL at the Unity level, but LeakSanitizer aborts the process at exit with the same 240-byte makeInstance->makeJob_ (jobj.c:768) leak signature, triggered via test_job_init_propagates_makeJobsub_failure (test_jobj_hardening.c:133). Confirmed this is the SAME pre-existing vendored-rtti-library characteristic already diagnosed in this entry's root_cause (1), not a new leak and not introduced by this feature's production code -- the feature's own new test is simply the first test in the suite to deliberately drive job_init() to fail after makeInstance() has already allocated, so it is the first to trip the latent characteristic. Still NOT FIXED (out of scope for a validation-only pass); flagging that under meson's full-suite invocation (abort_on_error=1) this leak turns into a SIGABRT / non-zero exit code for the lib test binary, so a naive exit-code-only CI gate would read this as a lib-suite failure even though zero Unity assertions fail and zero new markers/regressions exist in feature code.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1438.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1438"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-07-14T09:36:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/testsuite/test_commit_barrier_rollout.c"]
tags: ["job-array-single-wal-record", "wal", "test-helper", "offset-arithmetic", "deadlock", "futex"]
related: []
legacy: true
---
# bug-1438: test_job_array_submit_appends_exactly_one_wal_entry_and_one_continuation: Expected 11 Was 5 (WAL…
## Description
test_job_array_submit_appends_exactly_one_wal_entry_and_one_continuation: Expected 11 Was 5 (WAL type mismatch); subsequent test hung all 3 threads on futex_do_wait
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
jap_read_last_wal_hdr()/jap_read_last_wal_payload() were copied from read_last_wal_hdr() in test_ha_role_lifecycle.c, which computes the header offset as seg_offset - sizeof(hdr). That arithmetic is only correct for zero-payload entries (its only prior caller was a WAL_NOOP test). Every WAL_JOB_ARRAY_SUBMIT entry carries a nonzero capnp payload, so seg_offset - sizeof(hdr) landed inside the payload bytes, not the header -- read garbage decoded as a bogus type (5 == WAL_JOB_STOP). The first array test's TEST_ASSERT_EQUAL_UINT32 failure then longjmp'd out of the test body BEFORE reaching jap_teardown(), leaking that test's persist_t (background WAL-sync thread still running) into the next test's jap_setup() memset -- corrupting a live mutex mid-flight and deadlocking all 3 threads on futex_do_wait in the very next test.
## Fix
Replaced the back-computed-offset helpers with jap_read_wal_hdr_at()/jap_read_wal_entry_at(), which take the entry's start offset EXPLICITLY. Callers now snapshot g_jap.seg_offset immediately before each persist_submit_job_array_async() call and pass that snapshot in, instead of trying to infer the header position from the post-append offset.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

41
.wolf/bugs/bug-1439.md Normal file
View file

File diff suppressed because one or more lines are too long

39
.wolf/bugs/bug-1440.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1440"
status: "fixed"
status_updated: "2026-09-24"
status_note: "b7b90e9d"
component: "apiserver"
severity: "unknown"
timestamp: "2026-07-15T09:35:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/frontend/apiserver/backend/wire_dispatch_test.go"]
tags: ["cgo", "go-test", "toolchain-limitation", "wire_dispatch_test", "job-array-single-wal-record", "backend-package", "build-error", "golang-go-13938"]
related: []
fix_commits: ["b7b90e9d"]
legacy: true
---
# bug-1440: use of cgo in test wire_dispatch_test.go not supported / FAIL apiserver/backend [setup failed]
## Description
use of cgo in test wire_dispatch_test.go not supported / FAIL apiserver/backend [setup failed]
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Go toolchain hard restriction, not an environment/CGO_CFLAGS/nix issue: cmd/go's module indexer (GOROOT share/go/src/cmd/go/internal/modindex/read.go, ~line 589, inside the per-file import scan) explicitly checks `if imp.path == "C" { if isTest { badGoFile(name, fmt.Errorf("use of cgo in test %s not supported", name)) } }`. Any `_test.go` file that directly does `import "C"` is rejected outright the moment CGO is active for that file (CGO_ENABLED=1 and the file's build tag, here `//go:build cgo`, is satisfied) -- this is golang/go#13938, a long-standing toolchain limitation, confirmed by reading the actual installed Go 1.25.5 source in the nix shell. Because wire_dispatch_test.go and clib.go/convert.go/peers_test.go/handler_test.go/convert_test.go/heartbeat_test.go (the last four carry no build tag, so they build under BOTH cgo and !cgo) all live in the same package, the ENTIRE apiserver/backend package fails as `[setup failed]` under `make test-backend` (CGO_ENABLED=1) -- this also silently prevented the pre-existing cgo-buildable tests (handler_test.go, convert_test.go, peers_test.go, heartbeat_test.go) from running, even though none of them regressed on their own. The `!cgo`-tagged files (integration_test.go, control_host_test.go, stub_funcs_test.go) are unaffected since they're excluded from the CGO_ENABLED=1 file set entirely; `make test` (CGO_ENABLED=0) passed clean (apiserver/backend ok, 1.510s) confirming no regression on the non-cgo path.
## Fix
NOT FIXED -- user explicitly instructed build/run/report only, no source edits. Reported fix path for the user: remove `import "C"` (and any direct `C.xxx` calls) from wire_dispatch_test.go itself; move the cgo-dependent helpers (buildEncodedJobArrayPush, deliverViaRealDispatchChain, and the `/* ... */ import "C"` preamble with the wire_test_* static C helpers) into a new non-test `.go` file in the same package (e.g. clib_wire_test_helpers.go, `//go:build cgo`), exposing plain-Go-signature functions the _test.go file can call without itself using cgo directly. This mirrors the established pattern already used for other CGO test-only helpers in this package. Because the file was rejected before type-checking, it is unknown whether wire_dispatch_test.go has any further errors (undefined C symbols, type mismatches) beyond this first, blocking one -- those can only be discovered after this structural fix.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1441.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1441"
status: "fixed"
status_updated: "2026-09-24"
status_note: "wire_dispatch_support.go direct Go call"
component: "apiserver"
severity: "unknown"
timestamp: "2026-07-15T09:40:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/frontend/apiserver/backend/wire_dispatch_support.go"]
tags: ["cgo", "wire_dispatch", "go_push_handler", "cross-file-declaration", "job-array-single-wal-record", "test-backend", "CGO_CFLAGS"]
related: ["bug-1440"]
legacy: true
---
# bug-1441: backend/wire_dispatch_support.go:154:8: could not determine what C.go_push_handler refers to /…
## Description
backend/wire_dispatch_support.go:154:8: could not determine what C.go_push_handler refers to / FAIL apiserver/backend [build failed]
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
cgo declaration visibility is per-file, not per-package: go_push_handler is exported to C only via clib.go's own preamble comment ("extern int go_push_handler(xce_request_t*, void*);" immediately followed by "//export go_push_handler" on the Go func), and that extern declaration lives nowhere in any shared header -- it is a plain comment inside clib.go's /* ... */ block, not client.h/netio.h/request.h/controller.capnp.h. cgo processes each file's own preamble independently to build that file's C.xxx symbol table, so wire_dispatch_support.go (which #includes client.h/netio.h/request.h/controller.capnp.h but does NOT itself declare go_push_handler) has no visibility into that extern, even though both files are in the same Go package and both carry //go:build cgo. This surfaced only after fixing bug-1440's toolchain-level rejection (moving import "C" out of the _test.go file) and after separately discovering make test-backend also needs CGO_CFLAGS/CGO_LDFLAGS manually derived to match src/frontend/apiserver/meson.build (pkg-config --cflags/--libs for librtti, c-capnproto, openssl, tinylog, plus -I/src/include -I/src/build/lib -L/src/build/lib -lxce -pthread -lm) since the nix devshell itself does not export them and make test-backend invokes go test directly rather than through the meson/go_build path that normally injects them.
## Fix
NOT FIXED -- user explicitly instructed build/run/report only, no source edits. Reported fix path for the user: add the same "extern int go_push_handler(xce_request_t*, void*);" declaration to wire_dispatch_support.go's own cgo preamble (matching clib.go's exact signature), OR move that extern declaration into a shared header both files already include. Because the toolchain stops at first error, it is unknown whether wire_dispatch_support.go/wire_dispatch_test.go have any FURTHER errors (e.g. C.free_JobSubmit_ptr signature, C.extract_data, C.xce_encode/xce_decode field access) beyond this first blocking one -- those can only be discovered after this fix. Separately (env-only, no source change): make test-backend also requires CGO_CFLAGS/CGO_LDFLAGS to be exported manually before invoking make, mirroring meson.build's pkg-config-derived flags -- the bare Makefile command alone (as documented) does not produce a working CGO_ENABLED=1 build in this worktree.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1440

38
.wolf/bugs/bug-1442.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1442"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "apiserver"
severity: "crash"
timestamp: "2026-07-15T09:50:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/frontend/apiserver/backend/wire_dispatch_support.go"]
tags: ["cgo", "wire_dispatch", "asan", "double-free", "use-after-free", "job-array-single-wal-record", "test-backend", "go_push_handler", "free_Dataset", "free_Dataset_ptr"]
related: ["bug-1441"]
legacy: true
---
# bug-1442: AddressSanitizer: attempting double-free on 0x...1b0 in thread T0 -- fires immediately on…
## Description
AddressSanitizer: attempting double-free on 0x...1b0 in thread T0 -- fires immediately on TestWireDispatch_OnePushProducesNRecordsAndLogsOnce, aborts the whole backend.test binary before it can print PASS/FAIL, so TestWireDispatch_RedeliveryCreatesNoNewRecordsAndLogsNothing never runs at all
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
NOT ROOT-CAUSED (build/run/report only, no fix attempted per user instruction). bug-1441 (C.go_push_handler cross-file cgo resolution) is now fixed, and a separate missing `import "apiserver/store"` in wire_dispatch_test.go was fixed as a trivial compile error, so the build now succeeds and the test actually runs -- and immediately hits a real ASan double-free at runtime, not a build failure. Leading hypothesis (unverified): wireDispatchDeliver() (lines 122-180) calls go_push_handler(&decoded, nil) at line 164, whose defer C.free_Request(req) (clib.go) frees *decoded including its Request_body_datamsg payload -- the comment at lines 166-172 asserts free_Dataset() is 'a documented no-op for every table' (citing task 5.3b's residual-gap note) and therefore safe for wireDispatchDeliver to ALSO free ds.rows via free_JobSubmit_ptr() and ds itself at lines 173-177. A 16-byte double-free immediately after go_push_handler returns is consistent with that no-op assumption being wrong for the Data_data_jobArray-derived dataset table specifically (i.e. free_Dataset() actually does free ds->rows or ds itself for this table, and wire_dispatch_support.go's explicit cleanup then frees the same memory again). NOT CONFIRMED -- needs a symbolized ASan backtrace (frames are bare hex addresses in this build) or manual trace of free_Dataset()/free_Request() to confirm which free is the duplicate.
## Fix
FIXED and VERIFIED 2026-07-15T09:55:00Z. Root cause confirmed: go_push_handler's internal `defer C.free_Request(req)` (clib.go) calls free_Request(&decoded), whose Request_body_datamsg case calls free_Dataset_ptr(&decoded.body.datamsg) -- free_Dataset_ptr() (controller.capnp.c, generated) unconditionally free()s the dataset_t container pointer it is given, even though free_Dataset() itself is an empty no-op (never touches ->rows). So `ds` becomes a dangling pointer the instant go_push_handler returns -- the earlier NOT-ROOT-CAUSED hypothesis was correct in spirit but had the wrong culprit function (free_Dataset_ptr, not free_Dataset). wireDispatchDeliver() (wire_dispatch_support.go) had two bugs: (1) read ds.rows AFTER calling go_push_handler = use-after-free, (2) separately called C.free(ds) again = double-free on the same 16-byte region ASan caught. Fix: capture `rows := ds.rows` BEFORE calling go_push_handler (line 175), use that captured value afterward instead of re-reading ds.rows, and removed the redundant C.free(unsafe.Pointer(ds)) call entirely since go_push_handler's internal free_Request already frees it. Verified via full ASan CGO rebuild (make test-backend under nix develop ./src with CGO_CFLAGS/CGO_LDFLAGS = -fsanitize=address -g -DWITH_DEBUG=1 + pkg-config librtti/c-capnproto/openssl/tinylog): TestWireDispatch_OnePushProducesNRecordsAndLogsOnce and TestWireDispatch_RedeliveryCreatesNoNewRecordsAndLogsNothing both PASS with zero ASan errors; full ./backend/... suite (91 subtests, convert/handler/heartbeat/peers/wire_dispatch/auth) all PASS, zero regressions; make test (CGO_ENABLED=0) also fully green.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1441

39
.wolf/bugs/bug-1443.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1443"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-07-16T02:05:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/job.c (restore_job_array()", "bulk-insert phase)"]
tags: ["job-array-single-wal-record", "restore_job_array", "bulk-insert", "partial-visibility", "requirement-2.3", "requirement-9.4", "requirement-9.5", "requirement-9.6", "key_del", "kv.c", "kvmgr.c", "open-gap", "kiro-validate-impl"]
related: []
fix_commits: ["dcf9f39"]
legacy: true
---
# bug-1443: not a runtime failure report
## Description
OPEN, NOT A CRASH -- discovered as a design-time/test-proven gap, not a runtime failure report. restore_job_array()'s final bulk-insert pass into c->jobmgr can itself fail partway through (e.g. realloc() exhaustion growing c->jobmgr's backing table). Because the underlying xkvmgr_type store has no key_del, an already-inserted prefix [0, k) cannot be rolled back and is left permanently, genuinely visible in c->jobmgr -- individually addressable (status/cancel/completion all work via direct jobid lookup) -- while the function honestly returns a negative errno (never silently claims success). Critically, that prefix is never included in out_new_jobs, so HOOK_DETAIL_JOB_NEW never fires for it, not on this attempt or on any later successful retry of the same LSN (the per-index idempotent-skip logic on retry treats the already-inserted prefix as 'already exists', excluding it from the retry's own out_new_jobs too). Practical consequence: these array elements exist and are queryable forever, but never reach the scheduler's or API-server's push and so are never actually scheduled to run -- a silent, permanent no-op job, not merely a bookkeeping gap. requirements-coverage validation (kiro-validate-impl, 2026-07-16) confirmed this is a literal violation of Requirement 2.3 ('a standby shall never observe a single array WAL record in a partially-applied form') and Requirement 9.4 ('shall not construct a partial/divergent set of elements'), and its shadow extends to 9.5 (self-heal, since the orphaned prefix never gets a fresh chance to hook-fire even once the retry succeeds) and 9.6 (the capstone fault-injection test deliberately targets the fully-atomic construction-phase seam instead of this bulk-insert seam, precisely because asserting 'zero partial elements' against this seam would be dishonest).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
No capacity-reservation or rollback primitive exists anywhere in this codebase's kv.c/kvmgr.c/zarray.c stack -- c->jobmgr (xkvmgr_type) grows its backing table via realloc() with no pre-check, and once key_add() has inserted an element there is no key_del to undo it. restore_job_array()'s design (scratch-build every element first, then one bulk-insert pass under a single write-lock) makes construction-phase failures fully atomic (nothing ever touches c->jobmgr until every element is ready), but the bulk-insert pass itself is NOT covered by that atomicity guarantee -- it is the one place a late-stage resource-exhaustion failure can leave real, permanent state behind that the function's own success/failure contract cannot fully describe. Discovered and documented during task 3.1's original implementation review (job-array-single-wal-record spec), re-confirmed live-testable via test_rja_bulk_insert_failure_reports_error_not_false_success (test_job_array_apply.c), and re-surfaced by name during the spec's final /kiro-validate-impl requirements-coverage pass.
## Fix
FIXED and VERIFIED 2026-07-16 (commit dcf9f39, job-array-single-wal-record task 7.1, user-directed follow-up after the GO decision). Added kvmgr_reserve()/xzarray_reserve() -- new, purely-additive primitives in kvmgr.c/zarray.c following the existing kvmgr_capture_keys() template (bare extern function, no RTTI vtable slot, caller-holds-write-lock contract) -- that pre-grow c->jobmgr's backing xzarray for the whole array before the bulk-insert pass begins, closing the dominant N-scaling realloc-exhaustion trigger structurally. Discovered and fixed a genuine, previously-unknown off-by-one in the existing, unmodified xzarray_ensure_items(): its strict less-than sufficiency check (mincap < caps) means reserving exactly array_size and then performing array_size real stores still triggers one more growth cycle on the LAST store; reserving array_size + 1 is the exact minimal correct compensation, independently re-derived and confirmed with concrete arithmetic by adversarial review. For the now-much-narrower residual case (kvtab_add()'s own small per-entry hash-chain-node allocation, or a content-based -EEXIST collision -- both non-N-scaling, honestly still theoretically possible and documented as such, NOT claimed to be eliminated): restore_job_array() now populates out_new_jobs with whatever prefix was genuinely inserted before a mid-pass failure, and apply_committed_entry()'s WAL_JOB_ARRAY_SUBMIT case fires HOOK_DETAIL_JOB_NEW for that non-empty partial prefix even when the overall return is negative, while still propagating the negative rc so the LSN correctly halts and the Live Apply Resource Fence still engages for the remaining un-constructed indices -- a genuinely-inserted element is never left un-pushed/un-scheduled again. Verified: retry-double-notify risk traced and confirmed absent (pre-existing per-index idempotent-skip logic excludes already-inserted indices from any subsequent attempt's out_new_jobs); apply_wal_entry()'s dead-code standby path confirmed genuinely untouched; full controller/lib/scheduler test suites clean, zero regressions in any other consumer of the shared kvmgr.c/zarray.c code. Full detail in tasks.md's Implementation Notes, task 7.1 entry.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

39
.wolf/bugs/bug-1444.md Normal file
View file

@ -0,0 +1,39 @@
---
id: "bug-1444"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-16T02:48:18Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c (replay_wal()", "~line 3902-3904; consumed by persist_recover()/persist_verify_apply_correctness()", "~line 3960-3991)"]
tags: ["job-array-single-wal-record", "persist_recover", "replay_wal", "apply_verify_correctness", "APPLY_DIVERGENT", "boot-replay", "ha-availability", "non-promotable-fence", "xce-test-ha", "array-soak", "task-6.1", "raft-apply-decoupling", "ha-commit-durability"]
related: ["bug-1443"]
fix_commits: ["16747aa"]
legacy: true
---
# bug-1444: persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip…
## Description
persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip (expected last_applied_lsn=5 new_job_count=3); a hidden apply defect is present (ha-commit-durability recover_verdict=RECOVER_CLEAN) / persist_recover: UNRECOVERABLE -- post-replay apply-correctness verification is DIVERGENT ... Staying ALIVE and PERMANENTLY non-promotable (refusing to serve divergent state) and RETRYING catch-up indefinitely -- reproduced live during job-array-single-wal-record task 6.1's HA soak scenario the FIRST time a node is killed and restarted after ANY WAL_JOB_ARRAY_SUBMIT record has ever been committed to its local WAL (e.g. after a 2-element array submission).
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
replay_wal()'s post-replay apply-correctness expectation counter only tallies WAL_JOB_SUBMIT entries toward new_job_submits_out: `if (hdr.type == WAL_JOB_SUBMIT && new_job_submits_out && apply_rc == 1) (*new_job_submits_out)++;` (persist.c ~3902-3904). It has NO corresponding branch for WAL_JOB_ARRAY_SUBMIT (the new record type job-array-single-wal-record/task 1.1 introduced), even though apply_committed_entry()'s WAL_JOB_ARRAY_SUBMIT case (persist.c ~2809-2851, restore_job_array()) constructs N new job_t entries per record, not 1. So expected_new_job_count silently undercounts by N for every array record replayed during boot -- while persist_verify_apply_correctness() (persist.c ~3982-3988) measures actual_new_jobs via a REAL jobmgr entry-count delta (count_jobmgr_entries before/after), which correctly reflects the N elements. The mismatch (actual_new_jobs != expected_new_job_count) trips APPLY_DIVERGENT on every single restart of every node whose local WAL contains a WAL_JOB_ARRAY_SUBMIT entry above its checkpoint -- deterministic and 100% reproducible, not a race or resource-exhaustion condition. persist_recover_loud_fence_if_unrecoverable then classifies this as UNRECOVERABLE, permanently non-promotable-fencing that node for the rest of its process life (own boot-time verification re-runs identically on every subsequent restart too, since the miscounting bug reproduces every time). This is a NEW, separate gap from this spec's three self-flagged residual items (task 3.1 bulk-insert partial-visibility / bug-1443, task 4.2 boot-replay-doesn't-respect-apply-failures gap, task 5.3b's small per-commit leak) -- it lives in a DIFFERENT, pre-existing mechanism (raft-apply-decoupling task 5.1's post-replay apply-correctness verification / ha-commit-durability's Req 8.1-8.4), which this spec's own task list never touched to account for the new WAL_JOB_ARRAY_SUBMIT type. Practical severity: in any cluster where a job array has EVER been submitted, every ordinary node restart (routine maintenance, crash-restart, rolling upgrade) permanently non-promotable-fences that node; over enough restarts across a 3-node cluster this can leave ZERO nodes eligible for promotion -- a serious HA-availability regression, not a cosmetic log line. Discovered live via job-array-single-wal-record task 6.1's HA soak harness (.claude/skills/xce-test-ha/run-ha-soak.sh, new --array-soak mode) -- specifically NOT reproducible by this spec's existing white-box unit tests (task 2.3/3.4), which never drive a real persist_recover()/replay_wal() boot cycle after committing an array record.
## Fix
FIXED and VERIFIED (2026-07-15/16, commit 16747aa, task 6.0a of job-array-single-wal-record) -- independently root-caused by a separate implementation pass on the same branch, inserted as a blocking task ahead of 6.1/6.2/6.3 per explicit user sign-off after the exact same bug was live-reproduced during this task's own --array-soak run. Fix matches the shape predicted below almost exactly: replay_wal()'s tally gained a `WAL_JOB_ARRAY_SUBMIT` branch that adds `apply_rc` (the genuine per-record new-element count apply_committed_entry()'s array case already returns, not the flat/undercounted +1 the WAL_JOB_SUBMIT branch uses) to *new_job_submits_out when apply_rc > 0, mirroring the existing WAL_JOB_SUBMIT pattern. Verified via: (a) new unit tests in test_apply_correctness_verification.c with an independently-reproduced RED-before-fix/GREEN-after-fix cycle; (b) two from-scratch live 3-node HA soak re-runs (--array-soak), one by the fixing pass's implementer and one by its independent reviewer, both showing zero APPLY_DIVERGENT across all three of this bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-large-array) with one-WAL-record/one-commit-barrier and per-element status/cancel/completion all confirmed post-failover. Full detail in .kiro/specs/job-array-single-wal-record/tasks.md's Implementation Notes, task 6.0a entry. This buglog entry was originally filed (independently, before its author discovered the fix already existed on the branch) with fix=NOT FIXED; corrected here once the duplication was noticed, since the underlying finding is accurate and well-characterized and deserves its own tracked entry (task 6.0a's fix was documented in tasks.md but never separately logged here) -- kept as a distinct entry from bug-1443 (a different, still-open gap in the same feature) rather than merged or deleted.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1443

38
.wolf/bugs/bug-1445.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1445"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-17T09:47:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["job-array-single-wal-record", "ha.c", "promotion-gate", "live-apply-resource-fence", "boot-replay", "first-boot", "persist_recover", "replay_wal", "ordering-bug"]
related: ["bug-1443"]
legacy: true
---
# bug-1445: A node's very first promotion-eligibility evaluation in a fresh process runs before that node's…
## Description
A node's very first promotion-eligibility evaluation in a fresh process runs before that node's own first persist_recover()/replay_wal() call, so the Live Apply Resource Fence (job-array-single-wal-record tasks 4.1-4.3, extended to boot replay by task 7.2) is invisible to that one specific check. A node that would otherwise correctly fence itself non-promotable on a resource-exhaustion apply failure during its first-ever boot replay can instead pass its first promotion check looking fully eligible, because the fence flag hasn't been set yet at the point that first gate runs.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha.c's promotion-gate eligibility check (ha.c:1303-1408) runs before persist_recover()'s first-ever replay_wal() call (ha.c:1427-1439) on a freshly-started process. Every promotion evaluation AFTER a process's first one is correctly protected (persist_apply_resource_fenced() is threaded into both the periodic fence-release check and the election-time check per task 4.3) -- this is a narrow ordering gap specific to the single very-first evaluation of a brand-new process, discovered and documented (not fixed) during job-array-single-wal-record task 7.2's own review, out of that task's persist.c-only file boundary.
## Fix
FIXED and VERIFIED (2026-07-17, src/controller/ha.c). raft_cb_become_leader()'s is_first_boot block now re-checks persist_apply_resource_fenced(&c->persist) immediately after cluster_init_phase2() completes (the call that triggers this process's own first persist_recover()/replay_wal()) -- if that first replay just engaged the fence, promotion is refused (same pattern as the pre-existing refusal branch: WARN log + ha_arm_promotion_fence_timer() + return), instead of falling through to CTRL_PRIMARY on a decision made before that first replay ever ran. Every later election is unaffected -- is_first_boot is false by then, so the earlier eligibility gate (ha.c:1337) already covers it correctly. Scope check: confirmed ha_elect_evaluate_and_signal (the OTHER election path, ha.c ~5030) is dead in production for the promotion outcome -- every live call site of handle_request_vote_reply passes vote_granted=0 (bug-828 already redirected the real grant-recording path to the record-only ha_record_vote_reply, which never signals it) -- so raft_cb_become_leader is confirmed the sole live promotion path and this fix's scope is complete, not partial. TDD: new test test_first_boot_promotion_evaluates_own_first_replay_fence (src/controller/testsuite/test_job_array_replay_fence.c) wires ha_phase2_hook to a stub that calls persist_recover() for real, reproducing production ordering exactly (a never-recovered persist_t entering the election path with ha->phase2_done==0) -- RED-confirmed failing pre-fix at the exact predicted assertion (role_after_election==CTRL_PRIMARY), GREEN-confirmed passing post-fix. Full controller_unit_test binary (bundles every testsuite/test_*.c including test_convergence_verification.c, test_ha_cluster.c, test_p1_integration.c) green: 16/16 tests in this file, 0 regressions cluster-wide.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1443

40
.wolf/bugs/bug-1446.md Normal file
View file

@ -0,0 +1,40 @@
---
id: "bug-1446"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "build"
severity: "unknown"
timestamp: "2026-07-18T00:47:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/tools/test-commit-durability.sh (det_probe_once / date usage) -- test harness only", "NOT a product file"]
tags: ["test-harness", "false-positive", "busybox", "date", "kiro-validate-impl", "ha-commit-durability", "deterministic-mode"]
related: []
legacy: true
gitea_issue: 31
gitea_synced_status: "fixed"
---
# bug-1446: /kiro-validate-impl ha-commit-durability: test-commit-durability.sh --deterministic reported 3/3…
## Description
/kiro-validate-impl ha-commit-durability: test-commit-durability.sh --deterministic reported 3/3 FAIL ('DURABILITY GAP -- primary acked with 0 fdatasync in-window'), i.e. a false RED that looked like a regression of the spec's core invariant on the current build.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
This sandbox's /sbin/date is BusyBox date, which does NOT support GNU date's %N (nanoseconds) format directive -- 'date +%H:%M:%S.%N' literally emits the unexpanded string 'HH:MM:SS.%N' (confirmed by direct invocation). det_probe_once() captures t0/t1 with this call to bound the submit-to-ack window, then _hms_to_sod() converts them to seconds-of-day for the in-window fdatasync count. With the literal '%N' suffix the sub-second component is unparseable/degenerate, so the computed window fails to contain the real fdatasync timestamp even though it lands in the same wall-clock second -- producing syncs=0 (RED) regardless of actual ordering. This is an environment tooling gap (BusyBox vs GNU date), not a code regression: the script assumes GNU date is on PATH.
## Fix
NOT a product bug, no source fix applied (out of this validation's scope; script is owned by xce-test-e2e-primitives per this spec's task 1.3 boundary note). Diagnosed by cross-checking strace's own microsecond timestamps (which DO work correctly, e.g. 'fdatasync(10) = 0' at 00:47:01.128244) against the controller's committed_lsn advance log lines (commit_advance_locked '1 -> 2' at 00:47:01,144) for the same three probes: fdatasync consistently preceded the commit-advance (that gates the client ack) by 16-43ms in all 3 probes -- i.e. the real durable-before-ack ordering the spec requires DOES hold on the current build; the script's verdict was the false signal, not the product. Recommend the script detect/require GNU date (or use 1784335901.164264 / a C helper) rather than assuming %N support, so it fails loud on an incompatible date instead of silently producing a false RED.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Gitea: #31

47
.wolf/bugs/bug-1447.md Normal file
View file

@ -0,0 +1,47 @@
---
id: "bug-1447"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "crash"
timestamp: "2026-07-18T01:05:02Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/netio.c (handle_ssl_handshake / io_handler / io_worker -- crash site", "shared iomgr epoll dispatch); src/controller/ha.c (ha_connect_prep_peer/ha_connect_finish_peer/reconnect_lane_fn", "raft-drive-async Task 2.1/2.2 reconnect-offload -- suspected trigger); src/controller/cluster.c (handle_connection / endpoint_accept -- inbound accept path", "alternate suspect)"]
tags: ["ASAN", "FIXED", "SEGV", "endpoint_ref", "handle_ssl_handshake", "io_worker", "kiro-validate-impl", "netio.c", "on_standby_connected", "openssl", "raft-drive-async", "reconnect-offload", "regression-test", "round-9", "same-batch-uaf", "sequential-soak", "ssl3_read_bytes"]
related: ["bug-1282", "bug-1286", "bug-1298"]
legacy: true
gitea_issue: 32
gitea_synced_status: "fixed"
---
# bug-1447: /kiro-validate-impl raft-drive-async: 10-round SEQUENTIAL HA soak (the fully-supported…
## Description
/kiro-validate-impl raft-drive-async: 10-round SEQUENTIAL HA soak (the fully-supported single-failover lifecycle, hard-fail-on-any-loss) failed at round 9/10 -- 'FAIL: round 9: post-failover submit did not get 3 acks' (0/3 acked across 5 retry attempts). Root traced to ctrl-3 (a standby) crashing: AddressSanitizer DEADLYSIGNAL SEGV on address 0x40 (near-null, READ) in openssl's ssl3_read_bytes <- tls_get_message_header <- state_machine <- handle_ssl_handshake (src/lib/netio.c:548) <- io_handler (netio.c:588) <- io_worker (netio.c:633), thread T2 (an io_worker thread). With ctrl-1 already killed (the round's own kill target) and ctrl-3 now also dead, no 2-of-3 quorum remained for ctrl-2 (freshly elected PRIMARY, term=11) to commit writes -- explaining the observed 0/3-acks symptom as a downstream consequence, not the primary defect.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
CONFIRMED (deterministic regression test + live soak, not speculative). The earlier 'unsynchronized concurrent SSL_do_handshake from two io_worker threads' hypothesis is WRONG and is retracted: iomgr_start() creates exactly ONE io_worker thread per iomgr_t (src/lib/netio.c:iomgr_start), and a single controller process has exactly one such iomgr/thread for all peer connections -- there is no second io_worker thread to race against.
REAL mechanism: io_worker's epoll dispatch loop processes every ready fd from ONE epoll_wait() batch in a plain for-loop calling io_handler() per event (src/lib/netio.c, old io_worker ~line 612-640). A handler for an EARLIER event in that SAME batch can free a DIFFERENT endpoint that also has a ready event LATER in the SAME batch. In production this is exactly src/controller/ha.c:8257-8331 on_standby_connected(), the handshake-complete callback for a freshly-reconnected outbound standby channel (raft-drive-async Task 2.1/2.2's reconnect-offload path): when it detects a stale duplicate outbound channel to the same peer (ha.c:8286-8303, 'Different ep, same peer -> stale duplicate to evict'), it calls iomgr_unmonitor() (ha.c:8320-8321) then endpoint_unref() (ha.c:8327) on the STALE endpoint. iomgr_unmonitor()'s epoll_ctl(EPOLL_CTL_DEL) only stops a FUTURE epoll_wait() from returning events for that fd -- it does NOT invalidate an event for that SAME stale endpoint that is already sitting in the CURRENT batch's stack-local events[] array (very plausible during a reconnect burst, where several peer connections' fds become ready close together right after a new primary election). endpoint_unref() reaching refcnt 0 immediately SSL_free()s ep->ssl and free()s the endpoint_t. The loop's next iteration then calls io_handler() for that now-freed endpoint -- a heap-use-after-free that reads freed memory as 'ep', and (if it still looks like a plausible pointer chain) hands a freed/reused ep->ssl to SSL_do_handshake(), producing exactly the observed ASAN DEADLYSIGNAL SEGV deep inside OpenSSL's ssl3_read_bytes rather than an immediate top-of-function null-ssl crash.
This is a genuine gap in the bug-1282/1286/1298 hardening family: INVOKE_MSG_HANDLER (netio.h) already refs/unrefs an endpoint around its OWN msg_handler call (bug-1298 note in that macro's comment) so a handler cannot free the SAME ep it is currently dispatched on -- but that bracket never covered a DIFFERENT, already-batched endpoint torn down as a side effect (on_standby_connected's cross-endpoint eviction). io_worker's own epoll dispatch loop held no reference of any kind on the endpoints in its batch. Reproduced deterministically (no live soak needed to hit it) via a new unit test that drives io_worker's real batch-dispatch loop with a synthetic 2-event epoll batch where event 0's handler endpoint_unref()s event 1's endpoint -- confirmed RED pre-fix: ASAN heap-use-after-free at src/lib/netio.c:587 (io_handler reading ep->ssl_handshake_done), freed by src/lib/netio.c:843 (endpoint_unref) called from the injected handler, exactly mirroring the production on_standby_connected -> stale endpoint eviction -> later-batched io_handler() read chain.
## Fix
FIXED (2026-07-18). src/lib/netio.c: extracted io_worker's per-batch dispatch into a new static io_worker_dispatch_batch(struct epoll_event*, int) (~line 639-654), called from io_worker's while(1) loop in place of the old inline for-loop. It now takes an endpoint_ref() on every endpoint in the batch BEFORE dispatching any of them, dispatches all events via io_handler() as before, then endpoint_unref()s every endpoint only AFTER the whole batch has been processed -- mirroring the SAME endpoint_ref/endpoint_unref bracket idiom this codebase already uses for the vtask dispatch path (bug-1298) and INVOKE_MSG_HANDLER (netio.h), applied here to io_worker's own epoll dispatch loop, the one dispatcher that didn't yet have it. With this bracket, on_standby_connected's endpoint_unref() on a stale duplicate can only drop that endpoint's refcount from 2 (owner + io_worker's batch ref) to 1 during the batch -- never to 0 -- so the free is deferred until io_worker's own trailing unref runs after the ENTIRE batch (including that endpoint's own later event) has been safely dispatched. ref/unref are the existing lock-free _Atomic-int operations (netio.c endpoint_ref/endpoint_unref) with no I/O and no lock acquisition, so this adds zero blocking risk to the io_worker hot path (satisfies the async-io steering rule against blocking in vtask/event-loop handlers). Also added a small always-compiled test seam, io_worker_dispatch_batch_for_test() (netio.c + netio.h) -- not XCE_TESTING-gated because netio.c compiles once into libxce.a with no per-target defines (unlike controller/ha.c, which the controller test binary compiles directly with -DXCE_TESTING), so it follows this file's existing convention of unconditionally exporting its internals (endpoint_ref, endpoint_unref, endpoint_mark_tearing_down, etc. are all plain public functions already).
Regression test: src/lib/testsuite/test_netio_batch_dispatch_uaf.c (test_batch_dispatch_survives_cross_endpoint_eviction), registered in src/lib/meson.build and src/lib/testsuite/test.c. Drives io_worker_dispatch_batch_for_test() with a synthetic 2-event batch: event 0 -> an endpoint whose conn_handler calls endpoint_unref() on event 1's endpoint (simulating on_standby_connected's stale-duplicate eviction); event 1 -> an endpoint with refcnt==1 (its only ref is the one event 0's handler drops) whose conn_handler just records that it ran. TDD verified: RED confirmed pre-fix (ASAN heap-use-after-free at netio.c:587 in io_handler, object freed at netio.c:843 endpoint_unref from the injected handler -- captured verbatim before the fix was restored); GREEN confirmed post-fix (both handlers run safely, exit 0, re-confirmed again standalone after both live soaks completed). Full regression: libxce lib suite 294/0/0 (xcelib_test, ASAN, ~/xce.build), controller_unit_test 972/0/0 with zero AddressSanitizer/heap-use-after-free/heap-buffer-overflow/double-free/SEGV hits in the full test log. Live verification beyond the unit test: two independent fresh sequential 10-round /xce-test-ha HA soaks on real ext4 (/home/claude/tmp/xce-bug1447-verify-1: 10/10 rounds, 33 jobs, zero job loss, all 3 nodes led; /home/claude/tmp/xce-bug1447-verify-2: 10/10 rounds, 36 jobs, zero job loss, all 3 nodes led including a forced-election leadership-coverage check on ctrl-3) -- both fully clean, zero crashes, and an explicit grep of every node's logs in both runs for the bug-1447 signature (AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake) found zero matches.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1282, bug-1286, bug-1298
- Gitea: #32

40
.wolf/bugs/bug-1448.md Normal file
View file

@ -0,0 +1,40 @@
---
id: "bug-1448"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "build"
severity: "unknown"
timestamp: "2026-07-18T01:54:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/tools/xce-build.sh (skill invocation pattern", "not the script itself); .claude/skills/xce-build/SKILL.md's 'Always dispatch the build via the Agent tool' instruction"]
tags: ["xce-build", "Agent-tool", "run_in_background", "nohup", "setsid", "disown", "subagent-teardown", "kiro-validate-impl", "ha-heartbeat-committed-propagation", "process-detach"]
related: []
legacy: true
gitea_issue: 33
gitea_synced_status: "fixed"
---
# bug-1448: /kiro-validate-impl ha-heartbeat-committed-propagation: the xce-build skill's own documented…
## Description
/kiro-validate-impl ha-heartbeat-committed-propagation: the xce-build skill's own documented pattern ("dispatch via a plain Agent-tool call, run_in_background: false, wait for it") produced an INCOMPLETE build -- ninja log ends with 'ninja: build stopped: interrupted by user.' at 85/192 targets, roughly 14 minutes after launch with zero nix/meson/ninja processes still running and no error in the log. The synchronous Agent call returned to the caller almost immediately (~38s) with text claiming 'The build is running in the background... I'll wait for it to complete' -- i.e. the subagent itself spawned an internal Bash run_in_background job and then ended its own turn, rather than blocking until the command finished.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Same defect family as the already-documented xce-test-ha/run-ha-soak.sh 'context: fork' SIGHUP-teardown bug (fixed 2026-07-08), but manifesting through a DIFFERENT mechanism: xce-build's skill doc explicitly avoids 'context: fork' and claims a plain Agent-tool dispatch is safe -- but that safety claim implicitly assumes the subagent's own Bash tool call blocks synchronously for the whole build. In practice the subagent chose to run the long nix/meson/ninja command via its OWN internal run_in_background Bash call and then finished its turn, un-detached (no setsid/nohup/disown) -- so when that subagent's session/turn ended, the still-attached background job's process group got interrupted (ninja saw SIGINT/SIGHUP-equivalent, logged literally as 'interrupted by user'). A second SendMessage asking the same agent to 'wait for it now' resumed a session with nothing left alive to wait for.
## Fix
Relaunched the build via a fresh Agent dispatch whose ONLY instruction was to run `setsid nohup nix develop ./src --command bash src/tools/xce-build.sh controller asan </dev/null >LOG 2>&1 & disown` and return immediately (fire-and-forget), then polled LOG/binary-mtimes independently from the main session via the Monitor tool. That build completed cleanly in ~10 minutes (85/85 ninja targets, clean link, meson test suite started normally) with no interruption. General fix for any long shell command dispatched through the Agent tool that must outlive the dispatching subagent's own turn: always explicit setsid+nohup+disown, never rely on the subagent's internal run_in_background alone or on run_in_background:false at the Agent-tool-call level to guarantee synchronous blocking all the way through.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Gitea: #33

40
.wolf/bugs/bug-1449.md Normal file
View file

@ -0,0 +1,40 @@
---
id: "bug-1449"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "unknown"
severity: "unknown"
timestamp: "2026-07-18T09:56:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: [".claude/skills/xce-test-ha/run-ha-soak.sh (sequential mode round-1 kill timing) / src/controller/ha.c ha_on_primary_ctrl_fd_rdhup (ha-config-bootstrap Req 2.8 voter-only guard", "working as designed)"]
tags: ["ha-elect-ctx-lifecycle", "xce-test-ha", "run-ha-soak.sh", "ha-config-bootstrap", "test-harness", "flaky", "timing-race", "config-push", "round-1", "sequential-mode"]
related: []
legacy: true
gitea_issue: 34
gitea_synced_status: "fixed"
---
# bug-1449: [xce-test-ha] FAIL: round 1: no new primary elected after kill (no serving primary at term>2)…
## Description
[xce-test-ha] FAIL: round 1: no new primary elected after kill (no serving primary at term>2) -- hit once during ha-elect-ctx-lifecycle kiro-validate-impl soak, did NOT reproduce on immediate retry (10/10 clean)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Test-harness timing race, not a code defect. Root-caused from the controller logs (not assumed): cold-start election was unusually slow this run -- ctrl-1 only reached CTRL_PRIMARY (term=2) ~10.1s after process start (raft_cb_become_leader). Immediately on becoming primary it began its config-push handshake to both standbys (on_standby_connected: 'outbound standby connection to peer[N] established, ctrlhello sent', same ~10.1s timestamp). The driver's round-1 kill fired only ~1.7s later (its budget is 'submit 3 jobs + verify + walcompare' after primary is detected, no floor on how fast that completes) -- before either standby had received and applied the pushed config. Both standbys were still at config tag=0 when ctrl-1 died, so ha_on_primary_ctrl_fd_rdhup correctly refused to start PreVote/RDHUP re-election per ha-config-bootstrap's own Req 2.8 guard ('no local config (tag=0, n_peers=3) -- voter only, not starting PreVote/RDHUP re-election; awaiting configured primary'), leaving the cluster with no primary and no candidate. This is the guard working exactly as designed against an unconfigured standby self-electing -- the actual gap is that run-ha-soak.sh's sequential mode has no wait for initial 3-of-3 config convergence before round 1's first kill (unlike its wait for FULL reconvergence between later rounds). Sequential mode's round-1 timing has apparently never lined up this unluckily in any of the many other Group-1 kiro-validate-impl soaks run earlier the same session (ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote, raft-sim-conformance, libxce-failover, ha-entity-snapshots, ha-commit-durability, ha-term-safety, controller-status, job-array, job-array-single-wal-record) -- all of which use the same driver script unmodified and ran clean.
## Fix
FIXED (.claude/skills/xce-test-ha/run-ha-soak.sh). Added a round-1-only config-convergence gate that mirrors wait_for_full_convergence's exact polling style (same alive-count guard via pgrep -c -f "$(run_pattern)", same STABLE_TICKS=2-consecutive-converged-probes, same sleep 2 backoff, same ~90s/30-iteration bound, same log()/fail() convention) but gates on a NEW signal, cluster_config_converged (all 3 nodes report a non-zero configVersion via a new node_config_tag() single-node ctrlstatus client probe, mirroring node_probe's technique), instead of the existing cluster_committed_converged (committed-prefix catch-up) -- the two convergence signals are independent, and only the config-tag one covers the ha-config-bootstrap Req 2.8 voter-only guard. Added functions node_config_tag() / cluster_config_converged() / wait_for_initial_config_convergence() right after wait_for_full_convergence (~line 556). Call site inserted right after the chaotic-mode branch's closing `fi` and before the 'Steps 7-10 looped' round loop (~line 1897), so it applies only to the sequential/default path (chaotic mode intentionally tolerates this race by design -- 'no new primary' is a COUNTED outcome there, never a hard fail, so it was correctly left untouched). Diff: +77 lines, 0 lines removed -- no C source touched. VERIFIED: 3 back-to-back --sequential 6-round soaks on a genuine ext4 path (/home/claude/tmp, confirmed via `mount`/`df -T`, not overlayfs -- bug-924), all 3 PASS (6 rounds/24 jobs/zero loss/all 3 nodes led, identical verdict each run), and the new 'waiting for 3-of-3 initial config convergence before round 1' / 'confirmed' log lines appeared correctly ahead of every round 1. Cross-checked the actual cold-start timing from controller logs across all 3 runs to size the margin: election (raft_cb_become_leader -> CTRL_PRIMARY) took 10.3-11.7s after process start each run, and the config-push (handle_repl_config_push on both standbys) landed only 40-46ms after election every time -- i.e. the actual condition resolves in well under 100ms, while the new gate's minimum dwell is ~4s (2 ticks x 2s) and its ceiling is ~90s, giving roughly 2 orders of magnitude of margin over the observed timing, not just enough for these 3 runs' luck.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Gitea: #34

42
.wolf/bugs/bug-1450.md Normal file
View file

@ -0,0 +1,42 @@
---
id: "bug-1450"
status: "fixed"
status_updated: "2026-09-27"
status_note: "7aac018c: demotion-convergence offload ON by default; inline fallback when slow lane not started"
component: "controller"
severity: "unknown"
timestamp: "2026-07-18"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha_convergence.c", "src/controller/ha.h"]
tags: ["async-io", "thread-role-violation", "delaymgr", "slowlane", "ha_convergence.c", "demotion", "raft-drive-async-pattern", "xce-audit-invariants", "controller-tests"]
related: ["bug-1429", "bug-1451"]
fix_commits: ["7aac018c"]
legacy: true
gitea_issue: 25
gitea_synced_status: "fixed"
---
# bug-1450: xce-audit-invariants controller pass (dim: async I/O thread-role) found ha_demotion_convergence_fn…
## Description
xce-audit-invariants controller pass (dim: async I/O thread-role) found ha_demotion_convergence_fn scheduled directly as a delaymgr task body with no vtask/slow-lane hop: it truncates the WAL (persist_truncate_wal_above, file I/O under p->lock) then calls ha_self_restart (tlog_exit synchronous log flush + execv) -- all inline on the delaymgr thread, the SAME single thread that drives heartbeats and election timeouts cluster-wide for the whole cluster.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_demotion_convergence_fn (ha_convergence.c) ran its full body -- truncate + self-restart -- synchronously on whatever thread invoked it (the delaymgr worker in production). A prior fix attempt (2026-07-10, see bug-1429) tried moving this to a VTASK_LIVE_BLOCK vtask via c->ha.pool and deadlocked controller_unit_test, because c->ha.pool is never initialized/started by this file's lightweight unit-test fixtures (dc_setup), so a vtask enqueued VTASK_LIVE_BLOCK blocked forever waiting for a worker that never existed; that attempt was reverted.
## Fix
Split ha_demotion_convergence_fn into a dispatcher (fast admin work: single-flight CAS on new ha_state_t field demotion_convergence_inflight, then slowlane_submit) and a new demotion_lane_fn slow-lane worker body running the actual truncate+self-restart (extracted into ha_demotion_convergence_do_run) -- the SAME established slow-lane pattern already used for snapshot send / config-push / peer connect in ha.c (ha_snapshot_send_dispatch, ha_config_push_offloadable, ha_connect_to_peer), not the vtask pool that deadlocked before. New feature flag g_drive_async_demotion_offload (ha_set_demotion_offload_for_test / ha_demotion_offload_enabled, ha.h/ha_convergence.c) gates the offload. IMPORTANT: defaulting this flag to ON (1) was tried first and reproduced a NEW real hang -- raft_cb_become_follower (which schedules this task) is exercised by a very broad set of pre-existing controller_unit_test fixtures across many files that never call slowlane_init, so the offloaded work was silently queued on an uninitialized c->slowlane (queued, never run -- no live workers) and downstream code waiting on the now-never-happening completion hung. Reverted the default to OFF (0), matching the ORIGINAL rollout discipline of the three reference offloads (all shipped OFF-by-default, flipped ON only in a later dedicated compatibility-audit pass -- raft-drive-async task 3.4). The mechanism is correct and covered by dedicated tests (test_demotion_offload_task_returns_promptly / test_demotion_inline_task_blocks_RED in test_demotion_convergence.c) that explicitly flip the flag ON against a fixture with a real slowlane_init'd lane. Flipping the cluster-wide default to ON is a follow-up compatibility audit across the full test suite, explicitly out of scope for this fix. Verified: standalone controller_unit_test binary run directly (bypassing meson) reports 976 Tests 0 Failures 0 Ignored, no hang/timeout.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1429, bug-1451
- Gitea: #25

42
.wolf/bugs/bug-1451.md Normal file
View file

@ -0,0 +1,42 @@
---
id: "bug-1451"
status: "fixed"
status_updated: "2026-09-27"
status_note: "7aac018c: catch-up offload ON by default; inline fallback when slow lane not started"
component: "controller"
severity: "unknown"
timestamp: "2026-07-18"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/config_repl.c", "src/controller/ha.h"]
tags: ["async-io", "thread-role-violation", "delaymgr", "slowlane", "config_repl.c", "catchup-continuation", "raft-drive-async-pattern", "xce-audit-invariants", "controller-tests", "spurious-election-risk"]
related: ["bug-1429", "bug-1450"]
fix_commits: ["7aac018c"]
legacy: true
gitea_issue: 35
gitea_synced_status: "fixed"
---
# bug-1451: xce-audit-invariants controller pass (dim: async I/O thread-role) found ha_catchup_continuation_fn…
## Description
xce-audit-invariants controller pass (dim: async I/O thread-role) found ha_catchup_continuation_fn scheduled directly as a delaymgr task body with no vtask/slow-lane hop: catchup_stream_range -> ha_stream_wal_to_standby does blocking WAL file reads + socket writes to a standby directly on the delaymgr thread. Flagged as the MORE URGENT of the two violations -- catch-up can stream a variable, potentially large amount of WAL data, so it can block delaymgr (and therefore heartbeats to every OTHER peer) for an unbounded, unpredictable time, a real spurious-election risk.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ha_catchup_continuation_fn (config_repl.c) ran its full body -- re-derive target, then catchup_stream_range's blocking WAL reads/socket writes -- synchronously on whatever thread invoked it (the delaymgr worker in production). Source comment at config_repl.c confirmed a prior VTASK_LIVE_BLOCK vtask-pool attempt here (2026-07-10) also deadlocked controller_unit_test and was reverted (bug-1429), tracked as a known open violation.
## Fix
Split ha_catchup_continuation_fn into a dispatcher (fast admin work: slowlane_submit keyed by peer_idx+1) and a new catchup_lane_fn slow-lane worker body running the original streaming logic unchanged -- the SAME established slow-lane pattern used for snapshot/config-push/reconnect in ha.c, not the vtask pool. c->ha.catchup_inflight[peer_idx] (set by ha_walack_on_ack before scheduling) remains the authoritative per-standby single-flight guard; the lane's own per-key dedup is a redundant backstop, mirroring ha_connect_to_peer's reconnect offload. New feature flag g_drive_async_catchup_offload (ha_set_catchup_offload_for_test / ha_catchup_offload_enabled, ha.h/config_repl.c) gates the offload. Defaulting this flag to ON (1) was tried first and reproduced a real hang: test_convergence_verification.c's pre-existing ACK-driven catch-up fixtures (which directly exercise this exact code path via ha_walack_continuation_for_test) never call slowlane_init, so offloaded work was silently queued and never ran, hanging controller_unit_test at ~90s+ (confirmed reproducible both via meson and a direct standalone binary run with SIGQUIT timeout). Reverted the default to OFF (0), matching the same rollout discipline as bug-1450's demotion-convergence fix and the three original raft-drive-async reference offloads. Mechanism covered by dedicated tests (test_catchup_offload_probe_runs_promptly / test_catchup_inline_probe_blocks_RED in test_config_repl.c) using a REAL, started, single-worker delaymgr (delaymgr_init+delaymgr_start, not just an un-started fixture) plus a probe task queued behind the continuation to directly prove the delaymgr thread is/isn't blocked. Flipping the cluster-wide default to ON is a follow-up compatibility audit, out of scope for this fix. Verified: standalone controller_unit_test binary run directly reports 976 Tests 0 Failures 0 Ignored, no hang/timeout.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1429, bug-1450
- Gitea: #35

41
.wolf/bugs/bug-1452.md Normal file
View file

@ -0,0 +1,41 @@
---
id: "bug-1452"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-07-18T00:00:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/commit.c (crash site", "commit_advance_locked); src/controller/persist.c (fsync_worker background thread", "apply_committed_entry/restore_job/makeJob chain also appears in a related non-crashing stack from the same investigation); src/controller/ha_convergence.c + config_repl.c (g_drive_async_demotion_offload / g_drive_async_catchup_offload -- the flags whose flip triggers this)"]
tags: ["ha_convergence.c", "config_repl.c", "commit.c", "persist.c", "test_persist_truncate.c", "teardown-ordering", "toctou-race", "commit_engine_fini", "persist_stop", "raft-drive-async-pattern", "xce-audit-invariants", "controller-tests", "ASAN", "SEGV", "FIXED", "root-caused"]
related: ["bug-1429", "bug-1450", "bug-1451"]
legacy: true
gitea_issue: 36
gitea_synced_status: "fixed"
---
# bug-1452: AddressSanitizer: SEGV on unknown address 0x000000000038 in commit_advance_locked…
## Description
AddressSanitizer: SEGV on unknown address 0x000000000038 in commit_advance_locked (controller/commit.c:208), called from commit_on_durable_advance (commit.c:341) <- fsync_worker (persist.c:399), a background ASAN thread started by persist_start (persist.c:723) inside test_persist_truncate.c's make_fresh_persist / test_truncate_above_resolves_pending_continuations_as_failed. Reproduced by temporarily flipping BOTH g_drive_async_demotion_offload and g_drive_async_catchup_offload (bug-1450/1451) from their default 0 to 1 and running the standalone controller_unit_test binary directly (976-test suite) -- crashed immediately, before completing the suite. Confirmed absent with both flags at their committed default of 0: the same binary/build (just reverted) completed cleanly, 976 Tests 0 Failures 0 Ignored, twice.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
ROOT-CAUSED 2026-07-19: the g_drive_async_demotion_offload/g_drive_async_catchup_offload flags were a RED HERRING -- this exact SEGV signature (NULL deref in the fsync_worker -> commit_on_durable_advance -> commit_advance_locked chain) was already independently observed on 2026-07-07, long before those flags existed (see memory: 'Pre-existing ASAN SEGV in test_persist_truncate Unrelated to Bug-1417 Work'). The flags only perturbed thread-scheduling timing enough to make a PRE-EXISTING, always-latent test-teardown race fire more reliably; they do not participate in the crash's causal chain at all. Actual root cause: in src/controller/testsuite/test_persist_truncate.c, the three continuation-related tests (test_truncate_above_resolves_pending_continuations_as_failed, test_truncate_noop_does_not_resolve_pending_continuations, test_truncate_concurrent_burst_resolves_above_cutoff_exactly_once) called `commit_engine_fini(&ce)` BEFORE `teardown_persist(p)` (which calls `persist_stop(p)`) -- the exact opposite of the contract documented in commit_engine_fini's own comment (commit.c:464-468: 'Callers are expected to have stopped the flusher (persist_stop) first'). Because persist_stop had not yet run, the fsync_worker background thread was still alive and could be mid-flight invoking the durable-advance callback (fsync_worker, persist.c:398-399, calls p->on_durable_advance(p->durable_cb_ctx) with NO lock held) concurrently with commit_engine_fini destroying ce->lock/ce->cond and, as its final step, setting ce->p = NULL. This produces a genuine TOCTOU data race with no synchronization between the two threads: commit_on_durable_advance's own guard (`if (ce == NULL || ce->p == NULL) return;`) can read ce->p as still non-NULL (racing ahead of or concurrent with commit_engine_fini's teardown), pass the guard, and then commit_advance_locked's SEPARATE, later re-read of ce->p (commit.c:208, `atomic_load(&ce->p->committed_lsn)`) observes the NOW-NULLED value written by the concurrently-running commit_engine_fini on the main thread -- producing a NULL-pointer dereference at exactly offset 0x38 (offsetof(persist_t, committed_lsn)), matching the crash address bit-for-bit. persist_set_durable_cb's unhook (called from commit_engine_fini) is 'defensive and cheap' per its own comment but is NOT sufficient on its own without persist_stop's pthread_join guaranteeing the flusher thread cannot be in-flight at all -- confirmed by the fact that persist_set_durable_cb existed and ran in this path the whole time and the crash still occurred.
## Fix
FIXED 2026-07-19 (commit pending): added an explicit `persist_stop(p);` call immediately before `commit_engine_fini(&ce);` in all three affected test functions in test_persist_truncate.c (previously commit_engine_fini ran first, then teardown_persist -> persist_stop ran second -- backwards from the documented contract). persist_stop joins the fsync_worker thread, so by the time commit_engine_fini tears down ce->lock/ce->cond/ce->p, no thread can be mid-callback referencing them. teardown_persist's own internal persist_stop(p) call becomes a safe idempotent no-op on the second invocation (fsync_thread_running is already 0 and wal_fd is already -1, so both of its internal branches skip). No production code changed -- this was purely a test-fixture ordering bug, not a bug in commit.c/persist.c itself. Verified: full ASAN controller_unit_test suite (976 tests) run 6 times total (1 via meson + 5 standalone direct binary runs per project convention of not trusting meson's aggregate alone) -- 976 Tests 0 Failures 0 Ignored on every run, zero ASAN reports (SEGV/UAF/etc.), all 8 test_persist_truncate.c tests PASS every run. NOTE for a future session: the identical commit_engine_fini-before-persist_stop ordering also exists in test_demotion_convergence.c (line 233 commit_engine_fini(&g_dc.commit) before line 235 persist_stop(&g_dc.persist), against a real persist_start'd fixture) -- same latent race, not yet observed to crash there (likely a narrower window against a static/global fixture rather than a per-test stack fixture), out of scope for this fix but worth a follow-up sweep of all commit_engine_fini call sites.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1429, bug-1450, bug-1451
- Gitea: #36

40
.wolf/bugs/bug-1453.md Normal file
View file

@ -0,0 +1,40 @@
---
id: "bug-1453"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-20T00:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["multiple — see root_cause"]
tags: ["dead-code", "tls", "ca-cert", "cleanup", "controller", "worker", "scheduler", "xcecli", "apiserver"]
related: []
legacy: true
gitea_issue: 26
gitea_synced_status: "fixed"
---
# bug-1453: Unnecessary CA certificate plumbing across C daemons and Go CLI — ca_cert_path fields, --ca-cert…
## Description
Unnecessary CA certificate plumbing across C daemons and Go CLI — ca_cert_path fields, --ca-cert flags, and CACert config are dead or misleading; TLS verification is SSL_VERIFY_NONE everywhere
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
XCE design uses server-side TLS only (encryption without peer certificate verification). The CA cert plumbing was intended for future P3 mTLS hardening but was never implemented: (1) C-side: zero SSL_CTX_load_verify_locations calls in entire codebase — ca_cert_path in xce/cluster.h:35 is set by controller/worker/scheduler but never loaded into any SSL_CTX; do_connect() production path in cluster.c:73 is a NULL-returning stub; (2) Go-side: HTTPS is disabled by default; xcecli uses trust-on-first-use pinning of the server leaf cert (not a CA) stored via a misleadingly-named CACert field; falls back to InsecureSkipVerify:true. Design docs (persistence-and-ha-design.md:1700) explicitly tag peer verification as optional P3 hardening, confirming current encryption-only posture is intentional.
## Fix
DONE. C-side: removed ca_cert_path field from xce_cluster_config_t (src/include/xce/cluster.h); do_connect()/xce_connect_fn_t dropped the ca_cert param (src/lib/cluster.c); removed ca_cert_path non-empty check from xce_cluster_config_validate(); removed --ca-cert flag from worker (src/worker/src/main.c) and scheduler (src/scheduler/src/main.c); removed ca_cert_path population from src/controller/main.c:279-281; updated src/lib/testsuite/test_cluster.c mock_connect signature + fixture. Go-side: deleted xcecli/cmd/auth/cert/{cert.go,parser.go} (self-signed cert TOFU-pinning helper); removed CACert field from config.go APIServerInfo and models/logon.go LogonResponse; removed SetRootCAs/loadRootCAs from common/request/request.go and certPath param from Get/Post/DeleteRequestFromAPIServer; updated all ~18 xcecli cmd/* call sites; updated cmd/auth/logon.go to drop the https cert-fetch block; removed TestSetRootCAs_*/TestBuildTLSLogic from request_test.go. Verified: full C build (all components, ASAN) 353/353 targets, meson test suite 9/9 OK; xcecli go build/vet/test all PASS; grep -rn 'CACert|SetRootCAs|loadRootCAs|ca_cert' src/ clean (only historical design-doc mentions remain in src/docs/*.md, left as-is).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Gitea: #26

41
.wolf/bugs/bug-1454.md Normal file
View file

@ -0,0 +1,41 @@
---
id: "bug-1454"
status: "fixed"
status_updated: "2026-09-24"
status_note: "94f50ce6"
component: "scheduler"
severity: "leak"
timestamp: "2026-07-20T00:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/scheduler/src/scheduler.c"]
tags: ["scheduler", "peers", "verified-working", "minor-ctx-leak", "nmasters"]
related: []
fix_commits: ["94f50ce6"]
legacy: true
gitea_issue: 37
gitea_synced_status: "fixed"
---
# bug-1454: Scheduler --peers correctly wired for initial controller discovery; minor issues: nmasters=1 (no…
## Description
Scheduler --peers correctly wired for initial controller discovery; minor issues: nmasters=1 (no EndpointRotator) and leaked ctx on HA path
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
VERIFICATION: --peers IS correctly parsed via xce_parse_peers() (main.c:129), stored via sched_set_cluster_cfg() (main.c:156), and iterated in sched_init() HA loop (scheduler.c:462-493) BEFORE sched_init — init ordering is correct. Narrowed findings: (1) nmasters=1 per attempt (line 468) — manual candidate loop instead of filling config->masters[] from all candidates so libxce EndpointRotator never engages. Functional but suboptimal. (2) Leaked apictx on HA path — xcecli_ctx_init at line 454 allocates sched->apictx, then HA loop at line 472 re-inits without xcecli_ctx_fini on the first allocation. On standby-failure branches (lines 480, 492) cleanup is correct. OUT OF SCOPE: runtime reconnect/rejoin after controller death belongs to .kiro/specs/component-failover-adoption/ (requirements + design approved, tasks 3.1/3.2 not yet approved).
## Fix
DONE (leak fix only). Fixed the confirmed defect: sched_init() called xcecli_ctx_init(&sched->apictx, config) unconditionally, then on the HA path re-called xcecli_ctx_init() inside the candidate loop without freeing the first ctx -> leaked one xcecli_ctx_t per sched_init() call. Fix: xcecli_ctx_fini(sched->apictx) added immediately before the HA loop starts (src/scheduler/src/scheduler.c), since the loop's first iteration always replaces it. Verified via full ASAN build: all 5 scheduler meson test suites + libxce suite pass, no regression. REJECTED the nmasters>1+EndpointRotator suggestion: memory obs 11825 (this session) confirms scheduler/worker deliberately use the candidates[]-loop path instead of filling config->masters[], so changing that would contradict the established design, not fix a bug. Runtime reconnect/rejoin gap remains out of scope, deferred to component-failover-adoption spec (unchanged from original triage).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Gitea: #37

41
.wolf/bugs/bug-1455.md Normal file
View file

@ -0,0 +1,41 @@
---
id: "bug-1455"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "scheduler"
severity: "unknown"
timestamp: "2026-07-20T00:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/scheduler/src/scheduler.c", "src/scheduler/src/main.c", "src/scheduler/include/scheduler.h"]
tags: ["scheduler", "peers", "ha", "config", "api-design"]
related: ["bug-1454"]
legacy: true
gitea_issue: 38
gitea_synced_status: "fixed"
---
# bug-1455: Scheduler --peers silently ignored: sched_init() memset(sched,0,...) wipes sched->cluster_cfg that…
## Description
Scheduler --peers silently ignored: sched_init() memset(sched,0,...) wipes sched->cluster_cfg that main.c populated via sched_set_cluster_cfg() just before calling sched_init(), so the HA candidate loop's candidates_count check always reads 0 and falls back to the hardcoded localhost:9990 single-host path regardless of --peers.
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
sched_set_cluster_cfg(&sched, &cluster_cfg) at main.c was called before sched_init(&sched, &config, handler); sched_init's first action is memset(sched, 0, sizeof(*sched)), which zeroes the cluster_cfg field just written. sched_init's own internal HA-vs-single-host branch reads sched->cluster_cfg.candidates_count immediately after, so it always sees 0. User-reported regression found via code review after bug-1454's leak fix landed in the same function.
## Fix
Restructured instead of patched around: sched_init() signature changed to take `const xce_cluster_config_t*` directly (assigned to sched->cluster_cfg AFTER the memset, inside sched_init, not before via a separate setter). Deleted sched_set_cluster_cfg() (no other callers). main.c hoists cluster_cfg/cluster_cfg_ptr to function scope and passes cluster_cfg_ptr (NULL or &cluster_cfg) straight into sched_init(). Also fixed an adjacent smell in the same HA loop: it used to mutate the caller's xce_client_config_t* config in place per candidate, leaving it in a surprising mutated state after sched_init returned — changed to a per-iteration local copy. Verified: full ASAN build (all components) 353/353 targets, meson 9/9 OK, manual scheduler testsuite 15/15, Go tests all pass. No test exercises sched_init's HA path with real --peers input (pre-existing gap, not introduced by this fix).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1454
- Gitea: #38

41
.wolf/bugs/bug-1456.md Normal file
View file

@ -0,0 +1,41 @@
---
id: "bug-1456"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "scheduler"
severity: "unknown"
timestamp: "2026-07-20T00:00:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/scheduler/src/scheduler.c"]
tags: ["scheduler", "ha", "dangling-pointer", "lifetime", "client-config", "regression-of-bug-1454"]
related: ["bug-1454", "bug-1455"]
legacy: true
gitea_issue: 39
gitea_synced_status: "fixed"
---
# bug-1456: Dangling pointer risk in sched_init() HA candidate loop: a later refactor (following bug-1454)…
## Description
Dangling pointer risk in sched_init() HA candidate loop: a later refactor (following bug-1454) changed the loop to build a per-iteration stack-local xce_client_config_t candidate_config and pass &candidate_config into xcecli_ctx_init().
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
xcecli_ctx_init() (src/lib/client.c) stores the raw pointer it is given as ctx->conf — it does not copy the struct. That pointer is dereferenced later, long after sched_init() returns, by client_failover.c runtime reconnect/rotation logic (e.g. ac->conf->masters[next]). When the candidate loop used a stack-local xce_client_config_t per iteration, ctx->conf pointed at a stack frame that no longer existed once sched_init() returned — a classic dangling pointer that only manifests later during HA reconnect/rotation, making it easy to miss in a build+unit-test pass.
## Fix
Reverted to mutating the caller-supplied xce_client_config_t* config parameter in place (config->nmasters, config->masters[0], config->master_ports[0]) instead of a stack-local copy. config is owned by main()'s stack frame, which lives for the whole process, so storing &config-derived state in ctx->conf is lifetime-safe. Added a comment in scheduler.c explaining why the stack-local-copy pattern is unsafe here, to prevent it from being reintroduced.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1454, bug-1455
- Gitea: #39

40
.wolf/bugs/bug-1457.md Normal file
View file

@ -0,0 +1,40 @@
---
id: "bug-1457"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-20T22:56:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/myapp.c"]
tags: ["controller", "startup", "single-node", "pre-existing", "myapp_run", "cluster_open_listener", "discovered-during-review", "incidentally-fixed"]
related: []
legacy: true
gitea_issue: 27
gitea_synced_status: "fixed"
---
# bug-1457: xce_controller with no --peers flag exits with -EINVAL from cluster_open_listener instead of…
## Description
xce_controller with no --peers flag exits with -EINVAL from cluster_open_listener instead of starting as single-node primary
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Discovered while independently reviewing clusterconfig-schema-split task 5.3 by building and running the actual pre-spec baseline (commit 94f50ce) for comparison. Single-node startup (no --peers) was ALREADY broken before this spec began: myapp_run() calls cluster_open_listener() before the primary config (cluster_config) is loaded from disk, so the pre-spec fallback branch (cluster_config->port, only reachable when cluster_config != NULL) always hit its else branch and returned -EINVAL, unconditionally, for every zero-flag single-node start. This invalidates part of clusterconfig-schema-split's premise (requirements.md's Introduction claimed cluster_config->port was 'the sole live port-sourcing mechanism for single-node deployments' and 'single-node controller startup SHALL continue to work exactly as it does today' — in reality it did not work today). Not caused by, or a regression from, clusterconfig-schema-split's changes — task 2.1's port-resolution rewrite (commit 1229f9a) incidentally fixed this as a side effect, since its hardcoded-default fallback no longer depends on cluster_config being loaded.
## Fix
RESOLVED -- incidentally repaired by clusterconfig-schema-split task 2.1 (cluster_open_listener now falls back to a hardcoded XCE_CONTROLLER_DEFAULT_PORT constant instead of an unloaded cluster_config->port), not as a dedicated fix for this entry. Live-verified post-fix: single-node startup with no --peers now correctly binds port 9990 and reaches CTRL_PRIMARY. No further action needed unless a future audit wants to correct the historical claim in clusterconfig-schema-split's requirements.md/design.md that pre-spec single-node startup was previously working — it was not, for an unrelated reason (myapp_run's listener-open/config-load ordering).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Gitea: #27

38
.wolf/bugs/bug-1458.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1458"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "build"
severity: "unknown"
timestamp: "2026-07-22T03:22:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/nixSupport/xce.nix"]
tags: ["nix", "xceadmin", "python", "pip", "xce-build", "admin-config-tool", "task-1.3"]
related: []
legacy: true
---
# bug-1458: tools/xce-build.sh: line 134: pip: command not found (inside nix develop ./src, running the new…
## Description
tools/xce-build.sh: line 134: pip: command not found (inside nix develop ./src, running the new `xceadmin` build case, exit 127)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
admin-config-tool task 1.3 added python3Packages.{pycapnp,pyyaml,jsonschema,click,pytest,ruff} to xce.nix nativeBuildInputs per brief.md's Nix Environment section, but that list (copied verbatim from brief.md) omits python3Packages.pip. None of those packages transitively pull in a `pip` executable on PATH in a nix devShell built this way, so `pip install -e .` in the new xce-build.sh `xceadmin)` case failed immediately with command-not-found the first time it was actually exercised end-to-end inside `nix develop ./src`.
## Fix
Added python3Packages.pip to xce.nix nativeBuildInputs alongside the other python3Packages.* additions. Verified via `nix develop ./src --command bash -c 'which pip; pip --version'`.
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

38
.wolf/bugs/bug-1459.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1459"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "build"
severity: "unknown"
timestamp: "2026-07-22T03:45:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/tools/xce-build.sh"]
tags: ["nix", "xceadmin", "python", "pip", "pep668", "externally-managed-environment", "xce-build", "admin-config-tool", "task-1.3"]
related: ["bug-1458"]
legacy: true
---
# bug-1459: pip install -e
## Description
pip install -e . inside `nix develop ./src` fails: "error: externally-managed-environment ... This command has been disabled as it tries to modify the immutable /nix/store filesystem" (PEP 668)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
The `xceadmin)` case (admin-config-tool task 1.3) ran bare `pip install -e .`/`pip install .` against the nix-provided python3 directly, matching brief.md's harness sketch. But nixpkgs' python3 site-packages lives under the read-only /nix/store, and nixpkgs patches pip to refuse writes there (PEP 668 externally-managed-environment guard) even with --break-system-packages in some configurations. brief.md's sketch was never actually exercised end-to-end against a real nix devShell before this task, so the guard was never hit.
## Fix
Changed the `xceadmin)` case to create a throwaway venv (`$XCEADMIN_DIR/.venv`, recreated fresh every invocation) via `python3 -m venv --system-site-packages`, which inherits the nix-provided pycapnp/pyyaml/jsonschema/click/pytest (so pip never needs to rebuild those heavy binary deps from PyPI) while giving pip a normal writable site-packages to install xceadmin itself into. All subsequent commands (pip install, pytest) now run via $VENV_DIR/bin/pip and $VENV_DIR/bin/python. Verified end-to-end for both the default (editable) and release (pinned) profiles.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1458

38
.wolf/bugs/bug-1460.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1460"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "xceadmin"
severity: "unknown"
timestamp: "2026-07-22T03:52:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/xceadmin/pyproject.toml"]
tags: ["xceadmin", "python", "packaging", "setuptools", "package-data", "wheel", "admin-config-tool", "task-1.3", "requirement-5.5"]
related: ["bug-1458", "bug-1459"]
legacy: true
---
# bug-1460: Under `xce-build.sh xceadmin release` (pinned, non-editable install), the installed xcecommon…
## Description
Under `xce-build.sh xceadmin release` (pinned, non-editable install), the installed xcecommon package in site-packages has no schema_files/ directory at all — controller.capnp/c.capnp/JSON schemas are silently dropped from the wheel
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
pyproject.toml had no [tool.setuptools.package-data] entry for xcecommon, so setuptools' default wheel build only includes *.py files. The xce-build.sh sync step correctly copies controller.capnp/c.capnp/*.schema.json into src/xcecommon/schema_files/ in the SOURCE tree, but `pip install .` (non-editable) builds a wheel from that source tree and, without package-data config, drops every non-.py file. An editable install (`pip install -e .`) never surfaces this because it resolves imports straight into the checked-out source tree, bypassing wheel packaging entirely — exactly the gap design.md's Testing Strategy 'Unit — schema bundling' note warned only a release-profile test run could catch (Requirement 5.5).
## Fix
Added [tool.setuptools.package-data] xcecommon = ["schema_files/*.capnp", "schema_files/json/*.schema.json"] to src/xceadmin/pyproject.toml. Verified: after `xce-build.sh xceadmin release`, site-packages/xcecommon/schema_files/ now contains controller.capnp, c.capnp, and json/*.schema.json, byte-identical to the synced source copies.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1458, bug-1459

38
.wolf/bugs/bug-1461.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1461"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "unknown"
timestamp: "2026-07-22T02:00:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/schema/controller.capnp"]
tags: ["capnp", "schema", "controller.capnp", "shadowing", "type-collision", "wal-replication", "snapshot-transfer", "admin-config-tool", "task-3.3", "xceadmin", "cross-component"]
related: []
legacy: true
---
# bug-1461: pycapnp / capn-c: setting UpgradeConfigRequest.sha256 or .bytes raises…
## Description
pycapnp / capn-c: setting UpgradeConfigRequest.sha256 or .bytes raises capnp.lib.capnp.KjException: 'expected reader.type == STRUCT [6 == 10]; Value type mismatch' -- and the generated C accessors (UpgradeConfigRequest_get_sha256 etc.) were typed Data_ptr instead of a real byte blob
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
controller.capnp declares `struct Data $C.mapname("xce_data_t")` at line 355 (a tagged union of OPCode_get query-result shapes: schedjobs/schedworkers/jobs/workers/resources/resvals/jobArray). Cap'n Proto resolves unqualified type names lexically within a file, so this local struct SHADOWS the Cap'n Proto builtin `Data` blob type for every later `: Data` field declaration in the same file -- confirmed against the real compiled schema via `capnp compile -o-` + `capnp decode`, and against the actual capnpc-c generated C output (fields typed `Data_ptr`/`XceData_ptr`, a struct-pointer wrapper, using `capn_getp`/`encode_Data_ptr`/`decode_Data_ptr`/`free_Data_ptr`, instead of the correct `capn_data` blob type + `capn_get_data`). Affected 7 fields across 6 structs, several HA-critical: UpgradeConfigRequest.sha256/.bytes, ReplConfigReply.sha256/.bytes, ReplConfigPushRequest.sha256/.bytes, ReplEnvelope.payload (WAL replication envelope), SnapshotTransfer's chunk payload, WalCatchupEntry.payload/.batchPayloads (WAL catchup), CtrlHelloRequest.key. Production C code had already worked around this for years via hand-rolled raw capn_ptr manipulation (see request.c's own comment: "Pull sha256 and bytes via raw capn_list8 access (NOT decode_Data which would corrupt via xce_data_t)") -- the bug was known-and-routed-around, never fixed at the root, because nothing before xceadmin's pycapnp-based dynamic reflection needed the generated codec/accessors to work correctly for these fields (the C raw-workaround functions never called them). Two fields, `getdata` (RequestReply's OPCode_get reply payload) and `pushmsg` (Request's generic OPCode_push payload), correctly and intentionally referenced the LOCAL struct Data all along -- their surrounding dispatch code (Data_data_jobArray/jobs/workers/etc. switches in apiserver.c/opget.c/sched.c/handler.c/xceapi.c/helper.c) depends on it.
## Fix
Renamed `struct Data` to `struct XceData` in controller.capnp (kept `$C.mapname("xce_data_t")` so the generated C typedef name is unchanged), and explicitly repointed `getdata`/`pushmsg` at `XceData` by name (since removing the shadow means bare `Data` now correctly means the builtin blob, which those two fields do NOT want). Updated every derived-identifier reference (`Data_ptr`->`XceData_ptr`, `Data_data_<variant>`->`XceData_data_<variant>`, `struct Data`->`struct XceData`, `read_/write_/new_/encode_/decode_/free_Data(_ptr)?`->the `XceData` equivalents) across 19 hand-written files spanning controller, lib, scheduler, and the Go apiserver/xcecli CGo bindings (generated controller.capnp.c/.h regenerate automatically, never hand-edited). Fixed 5 call sites in src/lib/request.c (extract_data_bytes's parameter + 4 local variables) that had been declared XceData_ptr but now legitimately receive capn_data from the now-correct UpgradeConfigRequest_get_sha256/_get_bytes/ReplConfigPushRequest_get_sha256/_get_bytes accessors -- a pure type-name swap with zero logic change, since Data_ptr/XceData_ptr and capn_data share the identical {capn_ptr p} memory layout (confirmed: request.c already had a comment noting this exact equivalence). Verified via full rebuild+test of lib (1/1), controller (3/3, including the job-array push regression tests), scheduler (15/15), and the Go apiserver+xcecli backends (all packages green), plus a direct pycapnp round-trip proving UpgradeConfigRequest.sha256/.bytes now encode/decode correctly. Unblocks admin-config-tool task 3.3 (xceadmin.push's push_config()).
## Verification
_Not recorded (migrated from buglog.json)._
## References
_Not recorded (migrated from buglog.json)._

42
.wolf/bugs/bug-1462.md Normal file
View file

@ -0,0 +1,42 @@
---
id: "bug-1462"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-23T03:04:50.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["ha", "config-replication", "late-joiner", "committed_config_tag", "raft-promotion", "auto-detected"]
related: ["bug-1060"]
legacy: true
---
# bug-1462: ctrl-3 conf/ empty after HA failover deploy at /tmp/xce-ha-deploy -- config never replicated, but…
## Description
ctrl-3 conf/ empty after HA failover deploy at /tmp/xce-ha-deploy -- config never replicated, but WAL/jobs replicated fine
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Post-election baseline for ha->committed_config_tag (raft_cb_become_leader, ha.c:1279, set to safe_tag from ha_compute_safe_committed_lsn, ha.c:744-802) is computed as the MAJORITY-SAFE FLOOR across granted voters' self-reported tags (median of sorted tags, same conservative math used for WAL safe_committed_lsn). On a cluster where no admin config push has ever happened live (deploy-ha.sh places cluster.bin on disk pre-boot, not via a live push), every peer voter reports tag=0 except the newly-elected primary itself (self_tag=nonzero, e.g. 860cfd4f59ef9381) -- so the majority-floor computation picks 0, even though the primary demonstrably has a real, valid, currently-loaded config. This 0-floor is CORRECT for WAL LSN safety (a leader must not assume unacknowledged entries are durable) but WRONG for config_tag: a node's own actively-loaded config is not speculative the way an unacknowledged WAL entry is.
Effect: the ONLY path that successfully pushes config after promotion is the one-shot, promotion-instant fan-out (ha.c:1583-1602) to vote endpoints still connected AT THAT EXACT MOMENT (ectx->votes[i].ep != NULL) -- it correctly bypasses committed_config_tag and reads myconf->config_version_tag directly. Any standby that is NOT connected at that instant (a late joiner reconnecting seconds later via on_standby_connected, or any future reconnect/ restart for the rest of the term) relies entirely on the ONGOING heartbeat-driven decision loop (ha_dispatch_repl_action_for_standby -> ha_evaluate_repl_action, ha.c:3633-3663 / 3194-3199: 'if standby_tag < committed_tag -> HA_REPL_PUSH_CONFIG'). Since committed_config_tag is permanently floored at 0 for the rest of the term (nothing advances it except a live admin xceadmin config push via config_commit_change -> push_config_to_standbys, config_repl.c:1524/1556-1558, which never happens in a normal deploy-ha.sh-deployed cluster), standby_tag(0) < committed_tag(0) is false forever -> HA_REPL_PUSH_CONFIG never fires for that standby. Meanwhile WAL/job replication is unaffected since it correctly tracks committed_lsn (a genuinely-advancing value, set from real job commits), not the broken config_tag floor -- exactly matching the observed symptom (jobs replicate, config does not).
Reproduced live: /tmp/xce-ha-deploy, peers ctrl-1=9901/ctrl-2=9902/ctrl-3=9903 (src/build.dbg/controller/xce_controller). ctrl-1 promoted primary at term=2 via PreVote failover (ha_compute_safe_committed_lsn logged safe_committed_lsn=0 safe_config_tag=0 'from 2 granted voters'). ctrl-2 (already-connected vote ep[0]) received handle_repl_config_push 45ms after promotion (one-shot path). ctrl-3 connected 3s later as 'late-joiner standby peer[2]' (ha.c on_standby_connected), got WAL-caught-up correctly (ha_process_standby_state: standby[2] converged: config_tag=0 applied_lsn=1 committed_tag=0 committed_lsn=1) but NEVER received a config push -- ctrl-1's full log has zero further push_config/repl_config lines after promotion, and ctrl-3's log stays stuck at 'CTRL_STANDBY -- waiting for primary to connect and push config'. ctrl-3/conf/ is empty on disk (no cluster.bin.*, no current pointer) while ctrl-1/ctrl-2 both have cluster.bin.860cfd4f59ef + current.
## Fix
FIXED 2026-07-23. Implemented candidate direction (a): added ha_promotion_config_tag_floor(safe_tag, self_tag) (ha.h/ha.c, declared beside ha_promotion_truncate_floor) returning safe_tag > self_tag ? safe_tag : self_tag -- mirrors bug-1060's WAL-truncation clamp pattern but for config_tag instead of LSN. raft_cb_become_leader (ha.c ~1279) now clamps committed_config_tag to this floor instead of the raw majority-safe tag, so a newly-elected primary's own actively-loaded config tag is never dropped back to a stale majority-of-zero baseline. RED test added: test_promotion_config_tag_never_drops_below_own_loaded_config (testsuite/test_ha_election.c), covers the exact bug-1462 repro (safe_tag=0, self_tag=0x860cfd4f59ef9381 -> floor=self_tag), the equal case, safe_tag-above-self_tag case, and fresh-boot (0,0)->0. Unit suite green 977/977, zero regressions. Live regression: 20-round sequential soak (xce-test-ha, ~/tmp/xce-ha-soak-seq) PASS -- 63 jobs, zero job loss, all 3 nodes (ctrl-1/ctrl-2/ctrl-3) cycled through leadership across 20 promotions, 9m00.8s wall-clock. 20-round chaotic (overlapping-kill) soak (~/tmp/xce-ha-soak-chaos) also PASS -- VERDICT mode=chaotic rounds=20 jobs=57 committed_divergence=0 committed_job_loss=0 excluded_bug854_discovery=8 excluded_bug924_fdatasync=0 result=PASS (the 8 excluded bug-854 hits are client primary-discovery misses during overlapping-kill windows, not server-side loss). Confirms the fix holds under both normal and double-fault-within-reconvergence-window failover.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1060

38
.wolf/bugs/bug-1463.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1463"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "xcecli"
severity: "unknown"
timestamp: "2026-07-23T06:37:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/frontend/xcecli/common/request/request.go", "src/frontend/xcecli/cmd/jobs/peek.go"]
tags: ["xcecli", "tls", "https", "dead-code", "build-break", "ca-cert-removal", "auto-detected"]
related: ["bug-1453"]
legacy: true
---
# bug-1463: xcecli build fails: ../common/request/request.go:143:32: undefined: tls / :144:13: undefined…
## Description
xcecli build fails: ../common/request/request.go:143:32: undefined: tls / :144:13: undefined: loadRootCAs -- FAILED: [code=1] frontend/xcecli/xcecli
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
Commit 94f50ce6 (bug-1453, CA-cert-plumbing removal) deleted the crypto/tls import, loadRootCAs, and SetRootCAs from request.go and stripped certPath from GetRequestFromAPIServer/PostRequestToAPIServer/DeleteRequestFromAPIServer plus their ~18 call sites, but missed GetJobOutput() (still referenced tls.Config{RootCAs: loadRootCAs(certPath)} with no import) and its sole caller cmd/jobs/peek.go, which still called apiClient.SetRootCAs(serverInfo.CACert) (SetRootCAs no longer exists anywhere) and passed serverInfo.CACert (the CACert field was already removed from APIServerInfo in config/config.go). Go compiles per-package, so once common/request failed to compile, the compiler never got far enough to surface peek.go's own two dangling references in the same build run -- they were latent until request.go compiled again.
## Fix
FIXED 2026-07-23. request.go: removed GetJobOutput's dead TLS block, dropped certPath from its signature (token, jobID, params). peek.go: removed the SetRootCAs call, updated the GetJobOutput call site to the new 3-arg signature. Also enabled NewAPIClient's previously-commented-out `c.Client.SetTLSClientConfig(&tls.Config{InsecureSkipVerify: true})` -- per user request (xcecli needs to actually connect to an https API server) and per bug-1453's own documented design intent ("server-side TLS only, encryption without peer certificate verification"), since prior to this no xcecli method set any TLS config at all, meaning ANY https:// apiserver URL (self-signed/auto-generated cert, the common case) would have failed Go's default cert verification -- this was broader than just GetJobOutput. Added crypto/tls import accordingly. Documented both the api-server (http+https) and controller (server-side-cert-only, no client certs) TLS postures as a durable rule in .kiro/steering/security.md's Transport section, replacing a now-stale line claiming TLS is handled at the network layer. Verified: full `all debug` rebuild green (355/355 ninja targets), frontend/xcecli/xcecli binary produced with fresh mtime, no new warnings, no test fallout (grep for GetJobOutput/SetRootCAs/loadRootCAs across *_test.go in xcecli returns nothing).
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1453

38
.wolf/bugs/bug-1464.md Normal file
View file

@ -0,0 +1,38 @@
---
id: "bug-1464"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "build"
severity: "unknown"
timestamp: "2026-07-23T09:50:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/tools/xce-build.sh"]
tags: ["build", "xce-build.sh", "meson", "xceadmin", "controller", "ha-soak", "workflow"]
related: ["bug-1486"]
legacy: true
---
# bug-1464: deploy-ha.sh FAIL: Error: xceadmin binary not found -- run 'xce-build.sh xceadmin' or 'nix build…
## Description
deploy-ha.sh FAIL: Error: xceadmin binary not found -- run 'xce-build.sh xceadmin' or 'nix build .#xcecore' first (after controller had just been rebuilt and xceadmin had been built successfully earlier in the same session)
## Environment
_Not recorded (migrated from buglog.json)._
## Reproduction
_Not recorded (migrated from buglog.json)._
## Root cause
setup_build() unconditionally does rm -rf "$BUILD_DIR" before every single-component build (controller, xceadmin, etc.), and all components share the same default BUILD_DIR=xce.build for the asan profile. Building controller after xceadmin (or vice versa) wipes the previous component entirely out of the shared build dir -- only the most-recently-built component survives. Only COMPONENT=all does one setup_build() + a single unfiltered meson compile, so it is the only invocation that leaves multiple components coexisting.
## Fix
No code change; workflow fix -- when a downstream task (HA soak deploy-ha.sh) needs both xce_controller/xce_test AND the xceadmin venv present simultaneously in xce.build, run '/xce-build all' (or '/xce-build all asan') once instead of separate '/xce-build controller' + '/xce-build xceadmin' invocations, since the second call's setup_build() rm -rf wipes what the first call produced.
## Verification
_Not recorded (migrated from buglog.json)._
## References
- Related: bug-1486

Some files were not shown because too many files have changed in this diff Show more