docs(buglog): bug-1922 fixed; bug-1917 follow-up; soak coverage for bug-1919/1920

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/bugs/bug-1917.md

@@ -11,7 +11,7 @@ occurrences: 1
 files: ["src/lib/xcectx.c", "src/lib/netio.c"]
 tags: ["race", "failover", "client", "ssl"]
 related: ["bug-1916"]
-fix_commits: ["431a6d70"]
+fix_commits: ["431a6d70", "f26a2d45"]
 gitea_issue: 342
 gitea_synced_status: "fixed"
 gitea_synced_hash: "a66ad5a922dd"
@@ -48,6 +48,8 @@ UPDATE (2026-10-03): Review of the bug-1916 fix (2026-10-03) adds a second hazar
 ## Fix
 `xce_ctx_reconnect` first calls `xce_ctx_quiesce_ep`: unmonitor the old fd (no further io_worker events), clear `ep->owner` under `ep->lock` (stale reply tasks are dropped; `vtask_entry` frees their packet), then wait until the endpoint's reference count is back at its base value (0 for the embedded client endpoint, 1 for a heap one), polling every 1 ms. The wait is bounded at 2000 ms: if the caller is itself a task of this endpoint its own reference never drains, and the reconnect then proceeds as before with a WARN ("reconnecting anyway") rather than deadlocking. `vtask_entry` now reads `ep->owner` under `ep->lock`.
 
+UPDATE (2026-10-08): Follow-up (2026-10-08, f26a2d45): a reconnect made from inside a task of the endpoint it re-establishes no longer waits out the 2 s bound (vtaskmgr tracks the endpoint whose VTASK_IO task the thread runs; xce_ctx_quiesce_ep returns at once in that case), and old-connection tasks queued behind it can no longer reach the new session: each VTASK_IO task records ep->reconnect_gen when queued and vtask_entry drops a task with a stale generation.
+
 ## Verification
 - Test: `test_reconnect_drains_old_connection_tasks` (new src/lib/testsuite/test_ctx_reconnect_quiesce.c): reconnect to a local listener, queue an old-connection packet via INVOKE_MSG_HANDLER -> handle_msg with the pool stopped, reconnect again while a helper starts the pool 100 ms in. RED before the fix (the stale task reached the context's request handler: expected 0 calls, was 1), GREEN and LSan-clean after.
 - Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
@@ -55,5 +57,7 @@ UPDATE (2026-10-03): Review of the bug-1916 fix (2026-10-03) adds a second hazar
 
 UPDATE (2026-10-05): Live observation (2026-10-05, bug-1736 E2E): the xce_test fake worker hit the bounded-wait path -- a vtask handler of its own connection timed out waiting for a reply queued behind it (-110), failed over through xce_ctx_reconnect, and the quiesce logged 'old connection still has 2 in-flight reference(s) after 2000 ms; reconnecting anyway' and proceeded, as designed; no crash or ASAN report.
 
+UPDATE (2026-10-08): Follow-up: test_reconnect_from_own_task_does_not_wait_or_leak (test_ctx_reconnect_quiesce.c) -- a reconnect from the endpoint's own task with a stale message queued behind it: RED before (took the full 2 s bound), GREEN after (under 500 ms, the queued stale message never reaches the handler), LSan clean; xce-build all asan 127/127 (2026-10-07).
+
 ## References
 - Related: none

.wolf/bugs/bug-1919.md

@@ -63,5 +63,7 @@ Two defects in `promotion_fence_timeout_fn` (src/controller/ha.c):
 - Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
 - Live: 4x `/xce-test-ha` sequential 20 rounds, 2026-10-04. Run 2 hit the exact path: ctrl boot RECOVER_TRUNCATED → promotion REFUSED (durable_floor=323, HWM=323) → 30 s later "recovery-truncated log healed by catch-up — durable_floor=338 (baseline=323)" + "promotion fence RELEASED", and the soak PASSed (all 3 nodes led). Runs 1 and 4 PASS (path not hit). Run 3 FAILed on an unrelated replication catch-up stall (logged separately, no fence/truncation in that run).
 
+UPDATE (2026-10-08): Repeatable live coverage (2026-10-08, 5cf92335): run-ha-soak.sh --truncated-fence breaks ctrl-1's WAL tail CRC (RECOVER_TRUNCATED boot), makes it win an election (promotion REFUSED), and requires 'healed by catch-up' + 'promotion fence RELEASED' and then leadership: PASS -- refused on attempt 1, fence released after 30 s (the re-check window), healed ctrl-1 led at term 9, zero job loss.
+
 ## References
 - Related: none

.wolf/bugs/bug-1920.md

@@ -75,5 +75,7 @@ Scan of the four 2026-10-04 soaks: 3 nodes in 2 runs had a misfiled seg-1. It st
 - Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
 - Live: 4x `/xce-test-ha` sequential 20 rounds, 2026-10-04: 4/4 PASS. The pre-recovery drop fired 14 times (3-5 per run), no "found NO entries" warning, and a scan of every node's WAL found every segment's first LSN equal to its filename with no LSN jumps. Before the fix, the same scan found misfiled seg-1 on 3 nodes in 2 of 4 runs. Longer runs, 2026-10-04: sequential 50 rounds PASS, chaotic 50 rounds PASS (zero committed divergence), sequential 30 rounds under 40 CPU spinners on 32 cores PASS. The drop fired 7/10/5 times, there was no "found NO entries" warning, the WAL layout was clean on every node, and there were no ASAN reports.
 
+UPDATE (2026-10-08): Permanent gate (2026-10-08, 5cf92335): the sequential, chaotic and truncated-fence soak modes now run wal-layout-check.py before PASS; it flags the pre-fix runs' misfiled seg-...0001 and passed a sequential 20-round soak (25 segments, 0 problems).
+
 ## References
 - Related: none

.wolf/bugs/bug-1922.md

@@ -0,0 +1,59 @@
+---
+id: "bug-1922"
+status: "fixed"
+status_updated: "2026-10-08"
+status_note: ""
+component: "harness"
+severity: "wrong-result"
+timestamp: "2026-10-08T02:32:58Z"
+last_seen: "2026-10-08T02:32:58Z"
+occurrences: 1
+files: ["src/harness/test.c"]
+tags: ["harness", "worker", "resync"]
+related: ["bug-1921"]
+fix_commits: ["c06798b2"]
+---
+
+# bug-1922: xce_test fake worker ignores jobStateQuery; each join waits out the 15 s reconciliation timeout
+
+## Description
+**Observed:**
+```
+controller, after an xce_test fake worker joins:
+  WARN resync_on_timeout: resync: worker fakew5 reconciliation round 1 timed out after 15000ms -- resolving so dispatch is not withheld indefinitely (Req 5.3 ...
+```
+On every worker join the controller sends `OPCode_jobStateQuery` (worker-rejoin-resync) and withholds dispatch to that worker until the `OPCode_jobStateReport` answer arrives. `xce_test`'s worker context has no case for `OPCode_jobStateQuery` in its request switch, so it never answers and the controller resolves the round only by its 15 s timeout.
+
+**Expected:** the fake worker answers the query like xcecd does (xcecd_resync.c): every queried job ID once, in order -- running if it is running one of its jobs, otherwise unrecognized -- plus an entry for each running job that was not queried.
+**Impact:** harness only: every fake-worker join (and re-join, bug-1921) delays dispatch to that worker by 15 s, and the reconciliation never sees the jobs the fake worker is really running, so the controller's resync logic is untested with fake workers.
+**Found by:** test (bug-1921 live verification, 2026-10-06)
+
+## Environment
+- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
+- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: asan (xce.build)
+- Code: scheduler-framework @ 6a42631b-dirty
+- Deployment: single controller + scheduler + apiserver (`run-resource-verify.sh --keep-running`), one `xce_test` fake worker
+
+## Reproduction
+1. `bash .claude/skills/xce-test-resources/run-resource-verify.sh --keep-running --dir <D>`
+2. `printf 'set masters localhost:9991\nset myname fakew\nworker\njoin\n!sleep 30\nquit\n' | xce_test --norc`
+3. `<D>/logs/ctrl-1.log` shows "reconciliation round 1 timed out after 15000ms" for fakew ~15 s after the join.
+- Frequency: deterministic, every fake-worker join, 2026-10-06
+- Evidence: not kept (session scratchpad logs)
+
+## Root cause
+`xce_test`'s worker-context request switch (`handle_startjob`/`handle_notify`/`handle_jobcontrol`/`handle_runtasks` in src/harness/test.c) had no case for `OPCode_jobStateQuery`, the worker-rejoin-resync query the controller sends on every worker join. The query was silently ignored, so the controller could only resolve its reconciliation round by the 15 s timeout (Req 5.3), withholding dispatch to the fake worker until then.
+
+## Fix
+- New `src/harness/fake_resync.{c,h}`: `fr_add_page` reassembles a paginated query (an empty query is one complete page; a page that overflows the announced total resets the accumulator); `fr_build_report` builds the report entries like xcecd's `xcecd_resync_build_report` -- every queried ID once, in order, `running` when the fake worker runs it, otherwise `unrecognized`, then each running job that was not queried (`queried = 0`). The fake worker keeps no finished-job retention, so a finished job is reported unrecognized.
+- `test.c`: `OPCode_jobStateQuery` -> `handle_jobstatequery`, which answers with `xce_job_state_report_req` (a non-blocking request with no reply to wait for, so it is safe in the handler context, unlike bug-1921's notifies).
+
+## Verification
+- Test: `test_fake_resync` (src/harness/testsuite/test_fake_resync.c, harness_unit_test): single-page and multi-page reassembly, empty query, overflowing page rejected, report = queried in order (running/unrecognized) + unqueried running extras. RED before (module absent: link failure), GREEN after, LSan clean.
+- Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-07).
+- Live: single controller + scheduler + apiserver (`run-resource-verify.sh --keep-running`), 2026-10-08: an `xce_test` fake worker joined, a job was dispatched to it, and the worker joined again (a round is armed only for a host with non-terminal jobs): the worker printed `job state query answered: 1 job(s)` and the controller logged no reconciliation timeout for it (before the fix: 'reconciliation round 1 timed out after 15000ms').
+
+## References
+- Related: none