diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 5db80201..5ad371af 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -12785,22 +12785,24 @@ "error_message": "Intermittent (~2 of 9 attempts) client hang in the synchronous jobsub (COMMIT_MODE_WAIT) path shortly after a fresh single-node election.", "file": "src/controller/persist.c or src/controller/commit.c (unrooted)", "root_cause": "NOT ROOT-CAUSED. Flagged during raft-apply-decoupling task 7.4's throughput benchmark work. Reproduced identically on both the pre-fix and post-fix (task 6.1 apply-pump) binaries — confirmed unrelated to that fix. Resembles but is not confirmed identical to the closed bug-1416 boot-race class (persist_recover() replay-verify racing the live apply pump / live-replication apply path).", - "fix": "[UPDATE 2026-08-16] Live investigation (single-node deploy via /xce-deploy-ha, controller_unit_test's own fix build): found a RELIABLE reproduction recipe the original ~2/9-sequential-attempts description never had, and narrowed the root cause to a specific mechanism, but did NOT land a fix -- reporting findings honestly per the explicit scope of this investigation pass.\n\nRELIABLE REPRO (not sequential -- CONCURRENT): deploy a fresh single-node cluster, poll the log for CTRL_PRIMARY, then fire 9 xce_test client processes (each: connect + one jobsub) SIMULTANEOUSLY (backgrounded, not sequential) immediately after election. Reproduced in 3 of 4 rounds (1-3 of 9 concurrent clients failing per round, closely matching the original ~2/9 ratio) -- 0/45 reproductions when the SAME 9 attempts were fired sequentially instead (one xce_test process at a time). Concurrency at connect time, not raw timing after election, is the actual trigger.\n\nSYMPTOM, precisely: the failing client's xcecli_jobsubmit() (client.c, wrapped in XCECLI_FAILOVER_CALL) returns rc>=0 (success-shaped, NOT a negative error) after exactly ~10.0-10.1s, with the output jobid=0 (never written) instead of the submitted JobId -- test.c's cmd_jobsub (bug-786's own guard) correctly catches this as 'no clean ack' rather than misreporting success, so no data is actually lost/corrupted -- but the client hangs ~10s before getting a clear answer, matching the bug's own description exactly.\n\nCONFIRMED CLIENT-SIDE, not server-side: cross-referenced the controller's own log (fresh --logfile per round) against the 9 concurrent attempts -- server-side WAL only ever shows 8 job-submit entries committed (lsn sequence has no gaps, no admission-cap rejection WARN, no error of any kind) when 9 concurrent clients attempted -- the failing 9th request never reaches handle_jobsubmit at all. Ruled out the obvious admission-cap suspect: commit.c's COMMIT_MODE_WAIT admission control (cap=n_vcpus-VTASK_LIVENESS_RESERVED=3 on this 4-vcpu deploy) never logged its 'WAIT rejected, admission cap exhausted' WARN in any reproduction -- 8 concurrent WAITs completing well under the cap is plausible given each takes <100ms server-side once received.\n\nTHE ~10s NUMBER, precisely sourced: xceapi.c's xce_jobsubmit() (the innermost call XCECLI_FAILOVER_CALL wraps) has a HARDCODED cbuf_deadline_10s() wait on cbuf_timedget waiting for a reply to land in the single-slot ctx->wbuf. Read cbuf_timedget itself (mylock.c) -- confirmed CORRECT, it returns -ETIMEDOUT (negative) on a genuine timeout, which xce_jobsubmit correctly propagates as a negative rc. So a bare cbuf_timedget timeout cannot by itself explain the observed rc>=0-with-jobid=0 shape; something in the retry/rotate/recover layer ABOVE it must be involved.\n\nNARROWED SUSPECT (not confirmed): client_internal.h's own documented CI1 hazard -- 'handle_disconnect may have injected a stale status into the single-slot ctx->wbuf ... xcecli_ctx_recover drains any pending entry with a non-blocking cbuf_tryget so the next attempt's cbuf_get cannot consume a prior-attempt reply' -- but this drain only runs when xce_failover_rotate() succeeds (client_failover.c:130-153, mirrored by xcecli_retry_next per its own doc comment: 'only recover when the rotate succeeded'). On this single-candidate cluster, rotate reconnects to the SAME (only) node via a real xce_ctx_reconnect() -- plausible this usually succeeds (ruling out the simplest 'rotate never succeeds on single-node, drain never runs' theory) but NOT independently verified under the exact concurrent-connect contention this repro needs; a transient reconnect failure specifically during the connection-establishment storm (9 simultaneous TLS handshakes against a listener seconds old) could leave a stale wbuf entry undrained, which a LATER retry's cbuf_timedget could then either (a) consume immediately (fast, not 10s -- doesn't match) or (b) never resolve at all if the drain path interacts badly with an in-flight send, leaving the single wbuf slot permanently empty until the full 10s deadline. Did not instrument (fprintf/gdb) the actual attempt count or wbuf state during a live failing run to confirm this mechanism specifically -- xce_test's own -vvv flag produced no additional client-side retry/rotate trace output (goes through a different, unwired logger in this CLI context), and adding real instrumentation would need a fresh lib rebuild + re-repro cycle not completed in this pass.\n\nNOT YET DONE: instrument xcecli_retry_next/xce_jobsubmit directly (attempt count, wbuf drain outcome, rotate rc, per-attempt elapsed) and re-run the now-reliable concurrent-burst repro to catch the exact mechanism in the act -- this is the recommended next step, using the repro recipe above (no gdb needed; the reproduction is now reliable enough for print-based debugging). Confirmed unrelated to raft-apply-decoupling task 6.1's apply-pump fix (reproduces identically on that spec's pre-fix and post-fix binaries per the original entry) and to this session's bug-1428/1426/1474/1472-remainder fixes (none touch xceapi.c/client.c/client_failover.c's client-side retry path).\n\n[Original entry]\nNOT FIXED. Worked around via retry for task 7.4's benchmarking purposes only. Needs a dedicated root-cause investigation with the same live-repro rigor bug-1416 got (gdb-attach via GDB_WRAP seam, or targeted debug logging around the commit-wait barrier right after a fresh single-node election).", + "fix": "[UPDATE 2026-08-16] Live investigation (single-node deploy via /xce-deploy-ha, controller_unit_test's own fix build): found a RELIABLE reproduction recipe the original ~2/9-sequential-attempts description never had, and narrowed the root cause to a specific mechanism, but did NOT land a fix -- reporting findings honestly per the explicit scope of this investigation pass.\n\nRELIABLE REPRO (not sequential -- CONCURRENT): deploy a fresh single-node cluster, poll the log for CTRL_PRIMARY, then fire 9 xce_test client processes (each: connect + one jobsub) SIMULTANEOUSLY (backgrounded, not sequential) immediately after election. Reproduced in 3 of 4 rounds (1-3 of 9 concurrent clients failing per round, closely matching the original ~2/9 ratio) -- 0/45 reproductions when the SAME 9 attempts were fired sequentially instead (one xce_test process at a time). Concurrency at connect time, not raw timing after election, is the actual trigger.\n\nSYMPTOM, precisely: the failing client's xcecli_jobsubmit() (client.c, wrapped in XCECLI_FAILOVER_CALL) returns rc>=0 (success-shaped, NOT a negative error) after exactly ~10.0-10.1s, with the output jobid=0 (never written) instead of the submitted JobId -- test.c's cmd_jobsub (bug-786's own guard) correctly catches this as 'no clean ack' rather than misreporting success, so no data is actually lost/corrupted -- but the client hangs ~10s before getting a clear answer, matching the bug's own description exactly.\n\nCONFIRMED CLIENT-SIDE, not server-side: cross-referenced the controller's own log (fresh --logfile per round) against the 9 concurrent attempts -- server-side WAL only ever shows 8 job-submit entries committed (lsn sequence has no gaps, no admission-cap rejection WARN, no error of any kind) when 9 concurrent clients attempted -- the failing 9th request never reaches handle_jobsubmit at all. Ruled out the obvious admission-cap suspect: commit.c's COMMIT_MODE_WAIT admission control (cap=n_vcpus-VTASK_LIVENESS_RESERVED=3 on this 4-vcpu deploy) never logged its 'WAIT rejected, admission cap exhausted' WARN in any reproduction -- 8 concurrent WAITs completing well under the cap is plausible given each takes <100ms server-side once received.\n\nTHE ~10s NUMBER, precisely sourced: xceapi.c's xce_jobsubmit() (the innermost call XCECLI_FAILOVER_CALL wraps) has a HARDCODED cbuf_deadline_10s() wait on cbuf_timedget waiting for a reply to land in the single-slot ctx->wbuf. Read cbuf_timedget itself (mylock.c) -- confirmed CORRECT, it returns -ETIMEDOUT (negative) on a genuine timeout, which xce_jobsubmit correctly propagates as a negative rc. So a bare cbuf_timedget timeout cannot by itself explain the observed rc>=0-with-jobid=0 shape; something in the retry/rotate/recover layer ABOVE it must be involved.\n\nNARROWED SUSPECT (not confirmed): client_internal.h's own documented CI1 hazard -- 'handle_disconnect may have injected a stale status into the single-slot ctx->wbuf ... xcecli_ctx_recover drains any pending entry with a non-blocking cbuf_tryget so the next attempt's cbuf_get cannot consume a prior-attempt reply' -- but this drain only runs when xce_failover_rotate() succeeds (client_failover.c:130-153, mirrored by xcecli_retry_next per its own doc comment: 'only recover when the rotate succeeded'). On this single-candidate cluster, rotate reconnects to the SAME (only) node via a real xce_ctx_reconnect() -- plausible this usually succeeds (ruling out the simplest 'rotate never succeeds on single-node, drain never runs' theory) but NOT independently verified under the exact concurrent-connect contention this repro needs; a transient reconnect failure specifically during the connection-establishment storm (9 simultaneous TLS handshakes against a listener seconds old) could leave a stale wbuf entry undrained, which a LATER retry's cbuf_timedget could then either (a) consume immediately (fast, not 10s -- doesn't match) or (b) never resolve at all if the drain path interacts badly with an in-flight send, leaving the single wbuf slot permanently empty until the full 10s deadline. Did not instrument (fprintf/gdb) the actual attempt count or wbuf state during a live failing run to confirm this mechanism specifically -- xce_test's own -vvv flag produced no additional client-side retry/rotate trace output (goes through a different, unwired logger in this CLI context), and adding real instrumentation would need a fresh lib rebuild + re-repro cycle not completed in this pass.\n\nNOT YET DONE: instrument xcecli_retry_next/xce_jobsubmit directly (attempt count, wbuf drain outcome, rotate rc, per-attempt elapsed) and re-run the now-reliable concurrent-burst repro to catch the exact mechanism in the act -- this is the recommended next step, using the repro recipe above (no gdb needed; the reproduction is now reliable enough for print-based debugging). Confirmed unrelated to raft-apply-decoupling task 6.1's apply-pump fix (reproduces identically on that spec's pre-fix and post-fix binaries per the original entry) and to this session's bug-1428/1426/1474/1472-remainder fixes (none touch xceapi.c/client.c/client_failover.c's client-side retry path).\n\n[Original entry]\nNOT FIXED. Worked around via retry for task 7.4's benchmarking purposes only. Needs a dedicated root-cause investigation with the same live-repro rigor bug-1416 got (gdb-attach via GDB_WRAP seam, or targeted debug logging around the commit-wait barrier right after a fresh single-node election).\n\n[UPDATE 2026-08-16, part 2] FIXED. Root-caused via static code trace (no gdb/instrumentation needed after all): XCECLI_FAILOVER_CALL's idempotent-replay rule (client_failover.c, xce_failover_classify -- 'a duplicate on a re-sent write means the replay guard already applied our write, treat as success') can normalize a retried resend that hits the server's EEXIST/inflight-collision rejection into overall success (rc==0) -- but that rejection reply carries NO payload (handle_jobsubmit's EEXIST/inflight-collision branches reply with RequestReply_data_nodata, job.c:1441-1467), so xce_jobsubmit() (xceapi.c) never populates its *jobid output on the winning (dup) attempt, and never did on the earlier failing attempt either (its raw status was already negative). Net: xcecli_jobsubmit() can return rc>=0 with *jobid left at the caller's initial value (0) -- matching the confirmed symptom exactly (rc>=0, jobid=0, ~10.0-10.1s: one full 10s cbuf_timedget deadline on the failing first attempt, then a fast dup-rejection reply on the resend).\n\nFIX (TDD): added jobsubmit_finalize_jobid(uint64_t* jobid, uint64_t submitted_jobid, int overall_rc) (src/lib/client_internal.h declaration, src/lib/client.c definition), called from xcecli_jobsubmit() right after XCECLI_FAILOVER_CALL returns. Sets *jobid = submit->jobid (the CLIENT-generated id already on the wire -- job ids are client-generated via idgen(); the controller commits/echoes this exact value verbatim on every genuine success reply, bug-786/job.c:1439) whenever overall_rc >= 0, otherwise leaves *jobid untouched. Deliberately placed at the jobsubmit-specific wrapper layer, not the generic classify/decide layer (which is correctly type-erased and has no concept of jobid) -- test.c's cmd_jobsub (bug-786's own gate, comparing returned jobid == locally-generated jobsubmit.jobid) was ALREADY the right verification signal and needed no changes. 3 new pure unit tests in test_libxce_retry.c (test_jobsubmit_finalize_populates_jobid_on_success/leaves_jobid_untouched_on_failure/null_jobid_safe) -- socket-free, following this codebase's established direct-state-simulation test style for I/O-adjacent logic (test_netio_on_connected.c's precedent) rather than mocking a live connection.\n\nLIVE VALIDATION: full ASAN rebuild (0 compiler errors; controller_unit_test 1101 tests, 2 pre-existing unrelated test_config_repl.c failures; libxce test suite 1/1 OK, all 3 new tests PASS). Deployed a fresh single-node cluster and ran the same concurrent-burst repro as the original investigation (fresh election -> 9 simultaneous xce_test jobsub clients) for 5 rounds / 45 total clients: 0 'no clean ack' failures, 45/45 clean acks with jobid matching the submitted id (vs. the original repro's 1-3/9 failures in 3 of 4 rounds pre-fix). Honest caveat: this is probabilistic/timing-dependent (like the original repro), so 45/45 clean is strong but not airtight proof the exact race window is unreachable -- the fix is correct by construction regardless (it closes the *jobid gap for EVERY path that can produce rc>=0, not just the one race that was reproduced), so residual risk is limited to a still-undiscovered SEPARATE mechanism producing the same symptom, not this one going unfixed.", "tags": [ "ha", "commit-engine", "election", "intermittent", - "unconfirmed", "raft-apply-decoupling-followup", "live-repro-found", - "concurrent-connect-trigger" + "concurrent-connect-trigger", + "fixed", + "root-caused", + "tdd-red-green" ], "related_bugs": [ "bug-1416" ], - "occurrences": 1, - "last_seen": "2026-07-09T21:15:00.000Z" + "occurrences": 2, + "last_seen": "2026-08-16T11:10:00Z" }, { "id": "bug-1426", @@ -13922,7 +13924,7 @@ "error_message": "Host admin-control status changes (hostctrl -c / hostctrl -o, i.e. HOST_CLOSE/HOST_OPEN via worker_ctrl()) never propagate to standby members' in-memory worker state once the target host has already been joined -- discovered while root-causing bug-1471's single-byte checkpoint divergence, but the defect itself is general and has nothing to do with checkpoints, config upgrades, or the config-upgrade-snapshot spec.", "file": "src/controller/worker.c worker_from_capnp (~827-856); src/controller/persist.c case WAL_WORKER_CTRL apply path (~3724-3786)", "root_cause": "worker_from_capnp() (worker.c:827-856) is invoked UNCONDITIONALLY by persist.c:3731 for every WAL_WORKER_CTRL entry, regardless of its hook_detail discriminator (JOIN, UPDATE, or DEAD all share this one apply path -- see persist.c:3766-3783's switch, which only controls which post-apply NOTIFICATION hook fires, not whether the mutation itself is applied). worker_from_capnp's own idempotency guard (worker.c:847-853): if key_find() on c->workermgr finds a worker with the decoded name ALREADY PRESENT, it returns -EEXIST immediately, WITHOUT applying any decoded field (including host.status) to the existing worker object. persist.c:3732-3733 treats any -EEXIST return as 'idempotent re-apply (replay / resync) -- no hook' and silently returns 0. This EEXIST short-circuit is correct for its originally-intended purpose (a duplicate/replayed JOIN entry for a worker that's already been created should indeed be a no-op), but WAL_WORKER_CTRL is also the opcode for HOOK_DETAIL_WORKER_UPDATE (worker.c:732-794's worker_ctrl(), which handles HOST_OPEN/ HOST_CLOSE admin commands, sets w->host.status, then fires exactly this same HOOK_AFTER_EVENT/HOOK_DETAIL_WORKER_UPDATE/WAL_WORKER_CTRL path at worker.c:786-787) and HOOK_DETAIL_WORKER_DEAD. For UPDATE/DEAD events, the target worker EXISTING is the NORMAL, EXPECTED case (you can only update or kill an already-joined worker) -- yet the same 'already exists -> skip entirely' logic applies, so the decoded status change is silently discarded on every member OTHER than the one that directly processed the original RPC. Confirmed via git blame: worker_from_capnp was introduced 2026-05-08 (commit 9b6a91a8, 'feat: implement restore_job/worker_from_capnp for WAL recovery'), predating config-upgrade-snapshot's spec work by ~2.5 months; no config-upgrade-snapshot task touches this function or the WAL_WORKER_CTRL apply case. Empirically confirmed via byte-precise capnp decode of two independent live-soak checkpoint sets (/tmp/xce-ha-configupgrade{,-v2}): ctrl-1 (which processed a `hostctrl -c` against a previously-joined synthetic host) has host.status=0x00500000 (XCE_WORKER_CLOSED|XCE_WORKER_CLOSED_BY_ADMIN) in its checkpoint's HostAdminState frame; ctrl-2/ctrl-3 (standbys) both show host.status=0x00000000 for the identically-named host entry -- proving the close never reached their live c->workermgr entries. Because a standby's OWN restart+full-WAL-replay hits the identical code path (persist.c's replay call site also routes through apply_committed_entry's same WAL_WORKER_CTRL case), a member that fully replays from LSN 1 would ALSO reconstruct the JOIN first (creating the worker) and then skip the later UPDATE (already exists) -- so this is not even a restart-repairable staleness; the divergence appears to be permanent for that member until directly patched or until some other mechanism re-applies the admin action. General severity: this affects ANY admin host-control action (`hostctrl -c`/`hostctrl -o`) against an already-joined host, unconditionally, cluster-wide -- not just this soak's synthetic cfgupgrade-host scenario.", - "fix": "FIXED 2026-07-25 (TDD: RED test first, confirmed failing against the pre-fix code, then GREEN). worker_from_capnp() (src/controller/worker.c) now decodes the WAL_WORKER_CTRL entry's hook_detail discriminator (the same 4 leading bytes it previously skipped without inspecting) BEFORE applying its existing-worker idempotency guard. When the target worker already exists: HOOK_DETAIL_WORKER_JOIN keeps the original -EEXIST no-op (a duplicate JOIN replay is genuinely idempotent -- unchanged, regression-guarded). HOOK_DETAIL_WORKER_UPDATE now REPLACES (not ORs -- worker_ctrl()'s HOST_OPEN uses assignment-style &= ~(...) to CLEAR admin bits, so OR would leave a re-opened worker stuck closed) the existing worker's XCE_WORKER_ADMIN_MASK bits with the decoded payload's admin-masked bits, under the worker's own write lock (mirrors restore_host_state's masking discipline, worker.c:487, with replace instead of OR semantics), then returns 0 so persist.c's WAL_WORKER_CTRL apply case (unchanged -- its existing '0 on any successful apply, -EEXIST only for genuine JOIN no-op' contract already handled this generically) falls through to fire the correct HOOK_DETAIL_WORKER_UPDATE notification instead of silently discarding the status change. HOOK_DETAIL_WORKER_DEAD is DELIBERATELY LEFT UNFIXED (still -EEXIST/no-op) -- investigated, not skipped: mark_worker_unavail()'s mutation (worker.c:661-679) is NOT confined to XCE_WORKER_ADMIN_MASK (sets XCE_WORKER_UNAVAIL + dead_since + resets hostattrs_t to 'unknown' + zeroes resource-value slots via set_host_slots), and critically, the live per-host resource-value array xce_worker_t.resvals (a zarray_t, what set_host_slots actually mutates and what scheduling reads) is NOT part of the Worker capnp wire schema at all -- only the separate resvals_to_sched/nresvals_to_sched fields are (confirmed by reading the generated decode_Worker/encode_Worker in controller.capnp.c and the schema in controller.capnp). A naive status-replace for DEAD would be an incomplete fix (would not zero standbys'/replayers' resource slots on a dead-worker event, a real correctness gap for POST-PROMOTION scheduling since the newly-promoted primary would read stale, non-zeroed slots) and needs its own investigation into whether/how to extend the wire schema -- filed separately as bug-1472-remainder below rather than rushed. Fix verified via 2 new + regression tests in test_apply_committed_entry.c (test_ace_worker_ctrl_update_applies_to_existing_worker, test_ace_worker_ctrl_update_replaces_not_ors_admin_bits, test_ace_worker_ctrl_join_duplicate_existing_is_noop -- the last one a regression guard proving JOIN's existing no-op behavior is unchanged) plus the full controller_unit_test suite: baseline (this worktree, immediately pre-fix) 1030 Tests 0 Failures, post-fix 1033 Tests 0 Failures (net +3, exactly the new tests, zero regressions). persist.c was NOT touched -- confirmed its WAL_WORKER_CTRL apply case (persist.c:3724-3786) already treats any non-negative worker_from_capnp() return as a successful apply generically (re-looks-up the worker by name and fires the hook_detail-keyed notification unconditionally), so returning 0 from the new UPDATE branch required no caller-side change.", + "fix": "FIXED 2026-07-25 (TDD: RED test first, confirmed failing against the pre-fix code, then GREEN). worker_from_capnp() (src/controller/worker.c) now decodes the WAL_WORKER_CTRL entry's hook_detail discriminator (the same 4 leading bytes it previously skipped without inspecting) BEFORE applying its existing-worker idempotency guard. When the target worker already exists: HOOK_DETAIL_WORKER_JOIN keeps the original -EEXIST no-op (a duplicate JOIN replay is genuinely idempotent -- unchanged, regression-guarded). HOOK_DETAIL_WORKER_UPDATE now REPLACES (not ORs -- worker_ctrl()'s HOST_OPEN uses assignment-style &= ~(...) to CLEAR admin bits, so OR would leave a re-opened worker stuck closed) the existing worker's XCE_WORKER_ADMIN_MASK bits with the decoded payload's admin-masked bits, under the worker's own write lock (mirrors restore_host_state's masking discipline, worker.c:487, with replace instead of OR semantics), then returns 0 so persist.c's WAL_WORKER_CTRL apply case (unchanged -- its existing '0 on any successful apply, -EEXIST only for genuine JOIN no-op' contract already handled this generically) falls through to fire the correct HOOK_DETAIL_WORKER_UPDATE notification instead of silently discarding the status change. HOOK_DETAIL_WORKER_DEAD is DELIBERATELY LEFT UNFIXED (still -EEXIST/no-op) -- investigated, not skipped: mark_worker_unavail()'s mutation (worker.c:661-679) is NOT confined to XCE_WORKER_ADMIN_MASK (sets XCE_WORKER_UNAVAIL + dead_since + resets hostattrs_t to 'unknown' + zeroes resource-value slots via set_host_slots), and critically, the live per-host resource-value array xce_worker_t.resvals (a zarray_t, what set_host_slots actually mutates and what scheduling reads) is NOT part of the Worker capnp wire schema at all -- only the separate resvals_to_sched/nresvals_to_sched fields are (confirmed by reading the generated decode_Worker/encode_Worker in controller.capnp.c and the schema in controller.capnp). A naive status-replace for DEAD would be an incomplete fix (would not zero standbys'/replayers' resource slots on a dead-worker event, a real correctness gap for POST-PROMOTION scheduling since the newly-promoted primary would read stale, non-zeroed slots) and needs its own investigation into whether/how to extend the wire schema -- filed separately as bug-1472-remainder below rather than rushed. Fix verified via 2 new + regression tests in test_apply_committed_entry.c (test_ace_worker_ctrl_update_applies_to_existing_worker, test_ace_worker_ctrl_update_replaces_not_ors_admin_bits, test_ace_worker_ctrl_join_duplicate_existing_is_noop -- the last one a regression guard proving JOIN's existing no-op behavior is unchanged) plus the full controller_unit_test suite: baseline (this worktree, immediately pre-fix) 1030 Tests 0 Failures, post-fix 1033 Tests 0 Failures (net +3, exactly the new tests, zero regressions). persist.c was NOT touched -- confirmed its WAL_WORKER_CTRL apply case (persist.c:3724-3786) already treats any non-negative worker_from_capnp() return as a successful apply generically (re-looks-up the worker by name and fires the hook_detail-keyed notification unconditionally), so returning 0 from the new UPDATE branch required no caller-side change.\n\n[UPDATE 2026-08-16] Tag hygiene: this entry's own UPDATE-case fix landed 2026-07-25 (commit 7547a8f1) but its tags were never updated off 'not-fixed'/'needs-owner'. The DEAD-case gap this entry deferred is bug-1472-remainder, separately FIXED and landed today in commit 45400c3a. Both halves of bug-1472 are now fixed and committed; closing out.", "tags": [ "worker.c", "persist.c", @@ -13934,16 +13936,15 @@ "standby-divergence", "pre-existing", "out-of-config-upgrade-snapshot-scope", - "needs-owner", "root-caused", - "not-fixed" + "fixed" ], "related_bugs": [ "bug-1471", "bug-1472-remainder" ], - "occurrences": 1, - "last_seen": "2026-07-25T20:15:00Z" + "occurrences": 2, + "last_seen": "2026-08-16T10:35:00Z" }, { "id": "bug-1473",