forked from lei.chen/myxce
fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool, which hardcodes VTASK_LIVENESS for every inbound peer-control packet including vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in persist_promotion_advance_committed) was reachable from that never-blocks task class -- same bug family as bug-1198. Removed the force-flush; round-3 (durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL: persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path and does bounded pread() -- lower risk (page-cache-hot) but strictly still blocking I/O; not safe to remove outright (reopens bug-1043-family job loss), needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask). bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop, lock-free) and routed both call sites through it. Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
parent
ea0ac81c92
commit
40f510fb14
25 changed files with 1617 additions and 226 deletions
147
.claude/skills/xce-audit-invariants/SKILL.md
Normal file
147
.claude/skills/xce-audit-invariants/SKILL.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
---
|
||||
name: xce-audit-invariants
|
||||
description: "Audit a component's documented design constraints/invariants, fix confirmed violations, and prove the fix with build+unit+live HA regression. Usage: /xce-audit-invariants [component] [soak-rounds]. Component: controller (default, richest invariant surface), or any src/<component> with a CLAUDE.md/steering invariant section."
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash, Agent, TaskCreate, TaskUpdate
|
||||
argument-hint: "[component] [soak-rounds]"
|
||||
---
|
||||
|
||||
# XCE Design-Invariant Audit & Fix
|
||||
|
||||
Runs the same loop used to close bug-1417/1418/1419/1430/1431/1432: read the
|
||||
documented invariants → audit code against them in parallel → fix confirmed
|
||||
violations → prove it with a full rebuild, the unit suite, **and** a live HA
|
||||
soak (both chaotic and sequential modes). Code review alone is not sufficient
|
||||
proof here — several of the bugs above were only found by a live soak after
|
||||
the code-level fixes already looked correct.
|
||||
|
||||
## Args
|
||||
|
||||
- `[component]` (default `controller`) — which `src/<component>` to audit.
|
||||
`controller` is the primary target: it is the only component with a rich,
|
||||
explicit invariant surface today (`.kiro/steering/ha-async-io.md`,
|
||||
`persistence.md`, `error-handling.md`, plus `src/controller/CLAUDE.md`'s
|
||||
Process-Lifecycle invariant and TLA+ Protocol Verification Checklist). If
|
||||
invoked against another component, first check whether it has an equivalent
|
||||
invariant section in its `CLAUDE.md` — if not, say so and stop rather than
|
||||
inventing invariants to audit against.
|
||||
- `[soak-rounds]` (default `10`) — rounds for the final HA regression gate
|
||||
(only meaningful for `controller`; clamped to 5-50 by `/xce-test-ha`).
|
||||
|
||||
## Step 1 — Gather the invariant sources (do not skip)
|
||||
|
||||
Read, in full, before auditing anything:
|
||||
- `src/<component>/CLAUDE.md` — component-specific invariants (for
|
||||
`controller`: the "Process Lifecycle" DESIGN INVARIANT block and the full
|
||||
"Protocol Verification Checklist" under the TLA+ section).
|
||||
- The steering files that apply — for `controller`: `.kiro/steering/ha-async-io.md`
|
||||
(thread-role contract: iomgr/delaymgr must never block inline), `persistence.md`
|
||||
(WAL/commit/durability invariants), `error-handling.md` (ReturnError/errnox
|
||||
pattern, no silent error swallowing).
|
||||
- `.wolf/cerebrum.md`'s Key Learnings and Do-Not-Repeat sections — prior
|
||||
audits already found real gaps between what these docs claim and what the
|
||||
code does (e.g. a comment claiming a delaymgr task "starts immediately"
|
||||
when `delay_task_ms` actually delays its first fire — bug-1432). Treat
|
||||
every invariant-adjacent comment as a claim to verify, not a fact.
|
||||
- `.wolf/buglog.json` — grep for the component's tag to avoid re-auditing
|
||||
something already closed, and to recognize a recurrence of a known bug
|
||||
family (e.g. `bug-828`/`bug-851`/`bug-1060` class committed-job-loss bugs).
|
||||
|
||||
## Step 2 — Parallel audit dispatch
|
||||
|
||||
Fan out one Agent-tool subagent per invariant *dimension* (not per file) so
|
||||
findings stay independently verifiable. For `controller`, the dimensions that
|
||||
have historically found real violations are:
|
||||
|
||||
1. **Async I/O thread-role** — does any iomgr/delaymgr callback do blocking
|
||||
work inline instead of dispatching to a `VTASK_LIVE_BLOCK` vtask?
|
||||
2. **Process lifecycle** — does any code path call `exit`/`abort`/`_exit`/
|
||||
`tlog_exit` as a *designed response* to an operational condition (corrupt
|
||||
WAL, lost quorum, failed catch-up) instead of fence-and-stay-alive?
|
||||
3. **Raft/TLA+ safety** — walk the Protocol Verification Checklist in
|
||||
`controller/CLAUDE.md` line by line against the current `ha.c`/`persist.c`
|
||||
(election safety, leader completeness / no-self-truncation, commit rule,
|
||||
commit ordering, durability gating, freshness gate).
|
||||
4. **Commit/apply decoupling** — is entity materialization (`apply_committed_entry`)
|
||||
ever reachable before an entry's LSN is `<= persist->committed_lsn`, on
|
||||
any ingest path (standby WAL receipt, catch-up, heartbeat)?
|
||||
5. **Error handling** — silent swallowed errors, missing `ReturnError`, or a
|
||||
fail-open where the invariant docs require fail-loud-and-fence.
|
||||
|
||||
Each subagent must report file:line evidence for every claimed violation —
|
||||
no speculative findings. Use the Agent tool directly (general-purpose or
|
||||
Explore subagent type); do not invoke the Workflow tool for this unless the
|
||||
user has explicitly asked for multi-agent orchestration in their own words.
|
||||
|
||||
## Step 3 — Triage and fix
|
||||
|
||||
- Dedupe overlapping findings across dimensions.
|
||||
- For each CONFIRMED violation, fix it following the established patterns in
|
||||
this codebase rather than inventing new ones:
|
||||
- Blocking-work-on-delaymgr → the vtask dispatch pattern (`vtask_alloc` +
|
||||
`vtask_init` + `vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK)`, with
|
||||
an inline-fallback + WARN on alloc/enqueue failure — never silently drop
|
||||
scheduled work).
|
||||
- Self-termination on an operational condition → fence-and-retry (set a
|
||||
sticky non-promotable flag, log loud, schedule a retry via `delay_task_ms`,
|
||||
stay alive) — see `ha_self_restart_enter_fence_and_retry` for the
|
||||
reference shape.
|
||||
- A `delay_task_ms`-scheduled task whose doc/comment claims to run
|
||||
"immediately" → verify against `delay_task_ms`'s real semantics (first
|
||||
fire is AFTER `delay_ms` elapses, not before) before trusting the
|
||||
comment; if genuine immediacy is required and the task body only does
|
||||
non-blocking work, call it once synchronously before arming the periodic
|
||||
timer.
|
||||
- After each fix (or small batch of related fixes), rebuild with
|
||||
`/xce-build <component>` and confirm 0 test failures before moving on —
|
||||
don't accumulate multiple unverified fixes.
|
||||
|
||||
## Step 4 — Live regression (mandatory, not optional)
|
||||
|
||||
Code-level correctness is not sufficient proof for `controller`/HA work —
|
||||
bug-1431/1432 in this project's history were invisible to unit tests and the
|
||||
initial code audit, and only surfaced under a live chaotic soak. Run, in
|
||||
this order, using the `/xce-test-ha` skill:
|
||||
|
||||
1. `/xce-test-ha [soak-rounds] <scratch-dir>` in **chaotic** mode
|
||||
(`--chaotic`) — the adversarial overlapping-kill stress mode. Must reach
|
||||
`result=PASS` (`committed_divergence=0` **and** `committed_job_loss=0`).
|
||||
2. `/xce-test-ha [soak-rounds] <scratch-dir>` in **sequential** mode
|
||||
(default) — the supported single-failover lifecycle, including the
|
||||
leadership-coverage maneuver (all 3 nodes must lead at least once). Must
|
||||
print the terminal `PASS:` line.
|
||||
|
||||
Launch each soak directly from the main session via Bash with
|
||||
`setsid nohup ... & disown` (never through a forked/backgrounded sub-agent —
|
||||
see the `/xce-test-ha` skill's own warning about SIGHUP killing the cluster
|
||||
mid-round), then poll the log periodically (`ScheduleWakeup`, 3-5 min
|
||||
intervals) rather than blocking synchronously — a 10-round soak takes
|
||||
several minutes per mode.
|
||||
|
||||
**On FAIL**, root-cause from evidence, not guesswork:
|
||||
- Pull the exact failing round's timestamps from the driver log, then grep
|
||||
the per-node controller logs (`<dir>/logs/ctrl-*.log`) around that window
|
||||
for the relevant subsystem (`committed_lsn`, `safe_committed_lsn`,
|
||||
`converged`, `commit_advance_locked`, etc.).
|
||||
- Distinguish a **true durability violation** (`committed_divergence>0`, or a
|
||||
job still missing at final settlement) from a **visibility-latency gap**
|
||||
(`committed_divergence=0`, job reappears by final settlement — still a
|
||||
hard FAIL per this project's philosophy, but the fix must close the
|
||||
*window*, never unsafely expose an unconfirmed old-term entry early just
|
||||
to make the check pass — see bug-1431/1432 for the reference shape of a
|
||||
correct fix).
|
||||
- Fix, rebuild, re-run the SAME mode that failed before re-running the other.
|
||||
Repeat until both modes PASS.
|
||||
|
||||
## Step 5 — Record and report
|
||||
|
||||
- Append every genuinely new root-caused bug to `.wolf/buglog.json` (id,
|
||||
error_message, file, root_cause, fix, tags, related_bugs) — check the
|
||||
existing entry count first so the new `bug-NNNN` id doesn't collide.
|
||||
- Append durable, non-obvious learnings to `.wolf/cerebrum.md`'s Key
|
||||
Learnings section (anchor the edit near an existing dated entry to avoid
|
||||
the "## Key Learnings" heading collision — the file has one per dated
|
||||
session block).
|
||||
- Append a one-line summary to `.wolf/memory.md`.
|
||||
- Final report to the user: violations found (with file:line), how each was
|
||||
fixed, unit test pass count, and both soak VERDICT lines verbatim. Do not
|
||||
claim "fixed" for anything the soak didn't actually exercise.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: xce-test-ha
|
||||
description: "End-to-end HA failover soak test for an XCE controller cluster: deploy 3 nodes, submit jobs, then repeatedly kill the primary and verify zero committed-job loss across 5-10 rounds. Usage: /xce-test-ha [rounds] [dir]"
|
||||
description: "End-to-end HA failover soak test for an XCE controller cluster: deploy 3 nodes, submit jobs, then repeatedly kill the primary and verify zero committed-job loss across 5-50 rounds. Usage: /xce-test-ha [rounds] [dir]"
|
||||
model: haiku
|
||||
---
|
||||
|
||||
|
|
@ -23,7 +23,7 @@ A green run is `PASS: N rounds, M jobs submitted, zero job loss` (exit 0).
|
|||
|
||||
## Arguments
|
||||
|
||||
- Optional `[rounds]` — number of kill/restart rounds (clamped to 5–10, default 6).
|
||||
- Optional `[rounds]` — number of kill/restart rounds (clamped to 5–50, default 6).
|
||||
- Optional `[dir]` — deploy directory for the 3-node cluster
|
||||
(default `/home/claude/tmp/xce-ha-deploy`). Maps to `XCE_HA_TOP` /
|
||||
`run-ha-soak.sh`'s positional dir arg. Relative paths are resolved to
|
||||
|
|
@ -250,7 +250,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, or `factstream` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
| `XCE_HA_JOBS` | jobs per submit batch | `3` |
|
||||
| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` |
|
||||
| `XCE_HA_FACTSTREAM_ROUNDS` | `--factstream` kill-mid-fact-stream rounds | `2` |
|
||||
|
|
@ -277,7 +277,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
5. **Initial workload** — drives `xce_test`:
|
||||
`set masters … / client / wait-primary / jobsub ×N / joblist (expect-job) / walcompare`.
|
||||
Every acked `JobId` is tracked.
|
||||
6–11. **Kill loop** (5–10 rounds): discover the live primary → `kill -9` it →
|
||||
6–11. **Kill loop** (5–50 rounds): discover the live primary → `kill -9` it →
|
||||
restart the node downed in the *previous* round (so the cluster returns to 3
|
||||
members) → wait for a new primary → `verify_no_loss` (expect-job every tracked
|
||||
JobId) → submit another batch → `verify_no_loss` again → `walcompare`.
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ This feature makes a configuration upgrade establish a **full, entity-bearing st
|
|||
- The Raft protocol (term/log/commit) — owned by the `Raft TLA+ Conformance` library series.
|
||||
- Controller voting-set / membership changes — `ha-config-bootstrap`.
|
||||
- **Adjacent expectations**:
|
||||
- **HARD dependency on `entity-bearing-snapshots`**: today snapshots are a 16-byte LSN sentinel carrying no entity state; the COW machinery (`xzarray_freeze`/`thaw`) is built but dormant. That work must land first.
|
||||
- **Dependency on entity-bearing snapshots — SATISFIED (2026-07-09) by `ha-entity-snapshots` (21/21 tasks, fully approved).** That spec landed `write_snapshot`/`load_snapshot` (persist.c) as a direct Cap'n-Proto entity-encode/decode of job and queue records at checkpoint time, restoring before advancing position — not the COW `xzarray_freeze`/`thaw` mechanism this line originally anticipated (that machinery remains dormant; `ha-entity-snapshots` took a different, already-shipped implementation path). This spec no longer needs to wait; verify `write_snapshot`'s actual on-disk entity format (persist.c, `ha-entity-snapshots` design.md) before writing this spec's own design so it builds on the real interface, not the COW one originally assumed here.
|
||||
- Anchors on the `(term, lsn)` configuration-change marker from `ha-term-safety`.
|
||||
- Reuses the existing `snapshot_transfer_s` snapshot-install transport (chunked, `(term, lsn)`-anchored, hash-verified, install-before-activate).
|
||||
|
||||
|
|
|
|||
|
|
@ -19,5 +19,5 @@
|
|||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": ["entity-bearing-snapshots"]
|
||||
"blocked_on": []
|
||||
}
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ This spec delivers the convergence-and-recovery slice of the umbrella contract.
|
|||
1. When a primary observes a higher term and is demoted, the controller shall stop persisting new state changes before continuing demotion, so that no further provisional entries are produced. *(umbrella 2.5)*
|
||||
2. When a primary is demoted, the controller shall remove from its log all entries above its committed LSN. *(umbrella 2.5)*
|
||||
3. When demotion truncation completes, the controller shall restart its own process so that in-memory state is rebuilt through boot recovery from the persisted committed state only, discarding all provisional in-memory state. *(umbrella 2.5, 2.7)*
|
||||
4. If the process self-restart cannot be performed, the controller shall terminate fatally rather than continue running with potentially divergent state. *(umbrella 2.5; locked constraint)*
|
||||
4. If the process self-restart cannot be performed, the controller shall terminate fatally rather than continue running with potentially divergent state. *(umbrella 2.5; locked constraint)* — **SUPERSEDED (2026-07-10):** "terminate fatally" (`abort()`) violated the project-wide never-self-terminate invariant (CLAUDE.md "Process Lifecycle") on an operational condition (`execv()` can fail on `ENOMEM`/`EMFILE`/a permissions change mid-upgrade — none of these are programming-invariant violations). The safety property this criterion actually protects — a divergent ex-primary never serves authoritatively — is preserved without dying: on `execv()` failure the controller now sets a sticky, never-live-cleared `ha->self_restart_pending` flag (checked unconditionally by `raft_cb_become_leader` / `promotion_fence_timeout_fn` alongside `apply_divergent`, `ha.c`) and retries the re-exec periodically via delaymgr (`ha_self_restart_enter_fence_and_retry`, `ha_convergence.c`). The node stays alive, permanently non-promotable, and self-heals the moment a retry succeeds — never requiring manual admin recovery. See `ha-async-io.md`'s own `SUPERSEDES (2026-06-25)` note for the precedent of revising a locked decision when a stronger mechanism (loud-fence-and-stay-alive) was established after the original spec closed.
|
||||
5. When a demotion occurs, the controller shall log the event with the truncation boundary (committed LSN and prior log tip) before restarting, so that operators can observe the discarded range. *(umbrella 2.5)*
|
||||
6. Where test mode is enabled, the controller shall invoke a substitutable restart seam in place of process re-execution. *(locked constraint — testability)*
|
||||
7. The controller shall be able to reconstruct its in-memory state from its persisted committed state, discarding any provisional state not present in the committed log. *(umbrella 2.7)*
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ The authoritative subject for the acceptance criteria below is **the XCE control
|
|||
- Moving snapshot send, configuration-push waiting, or peer connect off the iomgr/delaymgr threads — owned by `raft-drive-async` (P2 transport/timer drive hardening); this feature owns only the commit-wait drive conversion (P1).
|
||||
- Changes to the worker join/rejoin protocol itself — owned by `worker-join-refactor` / `worker-rejoin-resync`.
|
||||
- Asynchronous DNS resolution for peer connections.
|
||||
- Extract-from-hook / apply-after-commit ordering for queue-control and worker-control commands: `queue.c`'s `AFTER_EVENT` path and `persist_queue_hook_fun`/`persist_worker_hook_fun` still mutate in-memory state before firing the WAL append, the same double-append-class ordering this feature fixes for job submission and facts — un-fixed here (2026-07-09 validate-impl finding). Tracked as a follow-up, same shape as this feature's Requirement 1/2 but for the queue/worker command family.
|
||||
- **Adjacent expectations**:
|
||||
- This feature relies on the majority-commit and idempotency primitives already delivered by `ha-commit-engine` and `ha-commit-barrier-rollout`; it does not re-implement them.
|
||||
- This feature touches the same submit/commit path that `ha-term-safety` is concurrently wiring into the raft library adapter; the two must be sequenced/coordinated so neither reintroduces a term-blind or apply-before-commit regression.
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
{
|
||||
"feature_name": "raft-apply-decoupling",
|
||||
"created_at": "2026-07-03T03:32:11Z",
|
||||
"updated_at": "2026-07-03T04:35:00Z",
|
||||
"updated_at": "2026-07-09T21:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"phase": "implemented",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
|
|
|||
|
|
@ -51,6 +51,13 @@ The design therefore: (1) collapses every entry-ingest path into the one reconci
|
|||
- Any change to the snapshot install boundary contract → revalidate `ha-convergence-recovery` rejoin path.
|
||||
- Closing this gate (green soak) is itself the revalidation trigger that **unblocks the `raft-drive-async` production flag-flip**.
|
||||
|
||||
### Retroactive Boundary Crossings (2026-07-09 validate-impl finding)
|
||||
|
||||
Two tasks materially crossed the "Out of Boundary" lines above. Both are self-labeled in tasks.md, non-conflicting with the specs that inherited them, and confirmed by a later cross-task validation pass — documented here so a future validate-impl doesn't re-flag them as new findings:
|
||||
|
||||
- **Task 5.9** shipped a commit-durability fix (`persist_submit_wal_and_commit` extraction, hook-rc-clobber job-loss) — territory line 19/36 assigns to `ha-commit-durability`. Landed here as a "down payment" (tasks.md's own words) because `raft-apply-decoupling` had no tasks.md yet and this spec's Req 8 soak needed zero job-loss to close. `raft-apply-decoupling`'s own task 3.1/3.2 later built the canonical version on the same code path — confirmed non-duplicating successor, not a competing fix.
|
||||
- **Task 5.8** shipped an endpoint-refcount/quiescence mechanism — territory line 20 explicitly excludes ("`raft-drive-async`... endpoint-quiescence — out of scope and must not change"). In practice it became the shared foundation `raft-drive-async`'s own task 3.2/3.4a cite as the reason their UAF hazards are "closed for real" — both specs' authors acknowledged the dependency in their own tasks.md, making this real, coherent, bidirectional coupling rather than silent scope creep.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
{
|
||||
"feature_name": "raft-ha-core-safety",
|
||||
"created_at": "2026-06-27T03:15:23Z",
|
||||
"updated_at": "2026-06-27T04:30:00Z",
|
||||
"updated_at": "2026-07-09T21:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"phase": "implemented",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
> TDD is mandatory (project rule): add the failing test/scenario first, then the change (red → green → refactor).
|
||||
> Almost all work is in the single test client TU `src/controller/testsuite/test.c`, so tasks are sequential by design; only the soak conversion touches a different file. Behavioral tests are driven through `xce_test` itself (piped scripts), gating on its exit code.
|
||||
|
||||
- [ ] 1. Foundation: verdict, loop wrapper, ergonomics
|
||||
- [x] 1. Foundation: verdict, loop wrapper, ergonomics
|
||||
- [x] 1.1 Session failure tracking + process exit code + abort/continue + comment/blank handling
|
||||
- Add file-scope state (`g_session_failed`, `g_continue_on_error`, `g_last_status`) and a `preprocess_and_run(line)` wrapper used by the main read loop: ignore blank and `#`-comment lines; run normal lines via `execute_line`; update `g_last_status`/`g_session_failed` from the result; in default mode stop on first failure, under `continue-on-error` keep going. `main` returns `g_session_failed ? 1 : 0`. Extend `cmd_set` with a `continue-on-error` toggle.
|
||||
- Observable: a piped script with a bogus/failing command makes `xce_test` exit non-zero; an all-valid script exits 0; with `set continue-on-error` a failing line does not stop later lines but the final exit is still non-zero; `#`/blank lines are ignored.
|
||||
|
|
@ -16,7 +16,7 @@
|
|||
- _Boundary: VerdictLoopWrapper_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 2. Shell-escape
|
||||
- [x] 2. Shell-escape
|
||||
- [x] 2.1 `!<cmd>` runs a shell command and feeds `$?`
|
||||
- In `preprocess_and_run`, when the first non-whitespace char is `!`, run the remainder via `system()`; set `g_last_status = WEXITSTATUS(rc)` (non-zero on `system` failure); a non-zero status is a command failure (subject to 1.1's abort/continue + exit code).
|
||||
- Observable: `!true` succeeds and sets `$?`=0; `!false` fails the session and sets `$?`≠0; a script can `!kill ...` / `!bash start.sh ...` to drive node lifecycle.
|
||||
|
|
@ -24,7 +24,7 @@
|
|||
- _Boundary: ShellEscape_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 3. Auto-variable substitution
|
||||
- [x] 3. Auto-variable substitution
|
||||
- [x] 3.1 Substitute the fixed auto-vars `$?`, `$jobid`, `$jobcount`
|
||||
- Add `substitute_autovars(line)` (run before tokenizing in `preprocess_and_run`): replace whole-token `$?`→`g_last_status`, `$jobid`→`g_last_jobid`, `$jobcount`→`g_last_jobcount`. A reference to an unset `$jobid`/`$jobcount` fails the command (no garbage substitution). No other variable syntax is recognized.
|
||||
- Observable: after a command, `echo $?` prints the prior status; referencing `$jobid` before any successful `jobsub` fails that command rather than substituting an empty/garbage value.
|
||||
|
|
@ -32,7 +32,7 @@
|
|||
- _Boundary: AutoVars_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
|
||||
- [x] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
|
||||
- [x] 4.1 `cmd_jobsub` reports success only on a clean ack with a valid JobId
|
||||
- Gate the `jobsubmit ok: JobId: <id>` report on a successful reply status AND a well-formed JobId; on success set `g_last_jobid` (feeds `$jobid`, 3.2). Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id (`63-15--1-...`).
|
||||
- Observable: a submit that does not get a clean ack returns failure and does NOT set `$jobid`; `jobsub` never prints success with a malformed id; a successful submit sets `$jobid` to the acked JobId.
|
||||
|
|
@ -40,7 +40,7 @@
|
|||
- _Boundary: JobsubFix_
|
||||
- _Depends: 1.1, 3.1_
|
||||
|
||||
- [ ] 5. Job-count capture and assertions
|
||||
- [x] 5. Job-count capture and assertions
|
||||
- [x] 5.1 `cmd_joblist` sets `$jobcount`; extract a shared count/contains helper
|
||||
- Refactor `cmd_joblist` to expose an internal helper that returns the current job count and tests membership of a JobId; set `g_last_jobcount` after a `joblist`.
|
||||
- Observable: after `joblist`, `echo $jobcount` prints the number of jobs returned; the helper is reusable by the assertions.
|
||||
|
|
@ -60,7 +60,7 @@
|
|||
- _Boundary: Assertions_
|
||||
- _Depends: 5.1, 3.1, 4.1_
|
||||
|
||||
- [ ] 6. Wait for elected primary
|
||||
- [x] 6. Wait for elected primary
|
||||
- [x] 6.1 `wait-primary [timeout_ms]`
|
||||
- Add `cmd_wait_primary`: loop a cheap, failover-aware client op (the existing `joblist`/probe path via `active_ctx`) until it succeeds or the timeout elapses (monotonic clock); default timeout when the arg is omitted; return 0 on success, `<0` on timeout.
|
||||
- Observable: against a healthy cluster `wait-primary` returns immediately; after a primary kill it blocks until a new primary serves, then succeeds; with no primary within the timeout it fails.
|
||||
|
|
@ -68,7 +68,7 @@
|
|||
- _Boundary: WaitPrimary_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 7. Reference E2E driver
|
||||
- [x] 7. Reference E2E driver
|
||||
- [x] 7.1 Convert `test-commit-durability.sh` to use the primitives
|
||||
- Replace the prose-grep + sanitizer-filter logic with piped `xce_test` scripts using `wait-primary`, `jobsub`/`$jobid`, `expect-jobs`/`expect-job`, and `!`-shell-escape for the primary kill/restart; gate each phase on the `xce_test` exit code. Preserve the rolling-kill + zero-loss coverage; add a `!`-driven cold-restart-all + `expect-jobs` step.
|
||||
- Observable: the converted soak runs end-to-end and reports pass/fail via `xce_test` exit codes (no joblist prose parsing); rolling-kill zero-loss coverage is retained; the cold-restart step asserts all committed jobs recover.
|
||||
|
|
|
|||
|
|
@ -11,18 +11,22 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
|
|||
```bash
|
||||
/xce-build # all components + all tests
|
||||
/xce-build lib # libxce + lib tests
|
||||
/xce-build controller # lib + controller + controller tests
|
||||
/xce-build controller # lib + controller + controller tests + xce_test driver
|
||||
/xce-build scheduler # lib + scheduler + scheduler tests
|
||||
/xce-build worker # lib + worker (daemon only, no unit tests)
|
||||
/xce-build apiserver # lib + apiserver + xcecli + Go unit tests
|
||||
/xce-build xcecli # xcecli + Go tests
|
||||
```
|
||||
|
||||
`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.
|
||||
|
||||
Each invocation:
|
||||
1. `nix develop` into the hermetic shell
|
||||
2. `meson setup build --buildtype=debug -Db_sanitize=address` (fresh clean build)
|
||||
3. `ninja -C build` (compile all + test executables)
|
||||
4. Run registered meson tests
|
||||
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build)
|
||||
3. `ninja -C <dir>` (compile all + test executables)
|
||||
4. Run registered meson tests (skipped for non-`asan` profiles — see below)
|
||||
|
||||
**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
|
||||
|
||||
## ASAN Integration
|
||||
|
||||
|
|
|
|||
|
|
@ -42,12 +42,14 @@ Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send
|
|||
|
||||
The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.
|
||||
|
||||
**Vote-request send failure MUST still reach the reply tally (bug-1423/1424).** `raft_cb_send_vote_request` / `raft_cb_send_pre_vote_request` send RequestVote/PreVote to every OTHER configured peer (dead or alive) and feed replies back to the library via `vote_reply_dispatch`/`pre_vote_reply_dispatch` (→ `raft_on_vote_reply`/`raft_on_pre_vote_reply`). A `endpoint_connect`/`iomgr_monitor` failure (the normal outcome when a peer is down) is itself an outcome that must be dispatched as a "not granted" reply through that SAME production path — never silently dropped, and never routed through the legacy test-only `handle_request_vote_reply`. Without this, the library's `replies_received == n_peers - 1` fast path ("everyone answered, decide now") is structurally unreachable whenever any peer is down, and recovery falls back entirely on the raw per-campaign timeout — real-election observed 2+ minutes permanently stuck in `CTRL_ELECTING` in a 3-node cluster with one node down before this fix.
|
||||
|
||||
- **RequestVote + RequestVoteReply** over Cap'n Proto
|
||||
- **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
|
||||
- **Votes persisted atomically** with term increment (raft_cb_persist)
|
||||
- **Majority wins**: quorum = ceil(n_peers / 2)
|
||||
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828).
|
||||
- **WAL truncation rule (load-bearing — bug-828)**: the **ONLY** truncates are (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, and (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile — both run *before* the new leader commits anything. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`) — it lags the true commit point, so a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".
|
||||
- **WAL truncation rule (load-bearing — bug-828)**: every `persist_truncate_wal_above` call site must cut at a value that is provably safe — never at the lagging learned-committed watermark (`last_known_committed_lsn`), since it lags the true commit point and a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). Four production call sites, all audited safe (2026-07-09): (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile, both running *before* the new leader commits anything; (c) `ha_demotion_convergence_fn` (`ha_convergence.c`), which truncates to the node's own local `committed_lsn` on demotion-triggered restart — a safe, never-stale local watermark, not the cross-node ctrlhello field this rule warns about; (d) `persist_wal_write_replicated`'s same-LSN payload-conflict repair (`persist.c`, `truncate_wal_above_locked(p, lsn-1)`), which removes exactly one divergent tip entry before re-appending the authoritative one at the same LSN and restores `committed_lsn` verbatim afterward — never regresses the committed boundary. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".
|
||||
|
||||
**Key safety invariants**:
|
||||
- `currentTerm` is monotonic (never decreases)
|
||||
|
|
@ -59,7 +61,8 @@ The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed v
|
|||
Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
||||
|
||||
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
|
||||
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
||||
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
||||
- **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
|
||||
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
||||
|
||||
## Replication Consistency (Two-Tier)
|
||||
|
|
|
|||
|
|
@ -58,6 +58,12 @@ Result: a truncated/behind node cannot win an election or pass the promotion gat
|
|||
- **Never write `committed_lsn` to disk.** Raft keeps the commit index volatile; durability is about the **log**, not the commit index. The post-election safe-commit point is re-derived from voter replies (`ha_compute_safe_committed_lsn`), never from a stale ctrlhello.
|
||||
- **LSN 0 is a reserved sentinel** — never a real entry; the first append is LSN 1. The commit engine rejects `lsn == 0`.
|
||||
|
||||
## Apply Correctness: `APPLY_DIVERGENT` (materialized state vs. WAL, not durability)
|
||||
|
||||
A **separate** verdict from `recover_verdict_t` (above): `persist_apply_verdict(p)` (`APPLY_CLEAN`/`APPLY_DIVERGENT`) answers "does the live materialized state (job/queue tables) actually match what a clean replay of the durable WAL would produce" — durability says the *bytes* are safe; this says the *applied result* is trustworthy. Computed once, at boot, in `persist_recover()`: an entity-count check (WAL_JOB_SUBMIT entries scanned vs. live table growth) plus a position check against `p->last_applied_lsn` (the LOGICAL apply cursor — never the physical WAL-scan tip, which also advances on checkpoint-marker/unknown-type entries and so falsely triggers DIVERGENT on an ordinary crash-during-checkpoint, bug-1419).
|
||||
|
||||
**Unconditional, restart-only fence (deliberate — landed by `raft-apply-decoupling` task 5.1-5.3, closes bug-1418's promotion-gate liveness-deadlock family):** once `APPLY_DIVERGENT`, the promotion gate (`ha.c` `raft_cb_become_leader` and `promotion_fence_timeout_fn`) refuses `CTRL_PRIMARY` **unconditionally** — no peer/HWM/durable-floor condition may clear it live, since a divergent local materialization can't be trusted for a peer/catch-up comparison either. Only a fresh, clean `persist_recover()` replay clears it (`apply_verdict` has exactly two production write sites: `persist_init()`'s reset and `persist_recover()`'s boot-time computation — no live path resets it). **Do not "fix" an apply-divergent liveness stall by relaxing this gate** — the actual fix is eliminating the false-positive DIVERGENT verdicts at the source (a boot-time TOCTOU where a returning node's verify pass raced the live apply pump / live-replication apply path, bug-1416, closed via `persist_t.recovery_complete` + `persist_mark_recovery_pending` called before any network traffic is processed).
|
||||
|
||||
## Fail-Loud, Never Self-Exit
|
||||
|
||||
When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a **loud permanent fence**: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky `recovery_unrecoverable` flag, hold non-promotable, **stay alive, and keep retrying** — it self-heals when a peer returns. It MUST NEVER `exit`/`abort` as a designed response. The decision is the pure `persist_recover_decision` → `RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}`; the action is `persist_recover_loud_fence_if_unrecoverable` (no terminal). Authoritative statement: `error-handling.md` + `src/controller/CLAUDE.md` "Process Lifecycle".
|
||||
|
|
|
|||
|
|
@ -12191,20 +12191,22 @@
|
|||
"error_message": "ha-commit-durability task 6.1 (cold-restart-all real-fs reproduction, src/tools/test-commit-durability.sh on a genuine ext4 deploy dir, ~/tmp not overlay): run 5 of 5 reported `expect-jobs: expected 19, actual 20` after a full cold-stop-all + cold-start-all cycle \u2014 the cluster had ONE MORE committed job than the harness's own ACKED_FILE (deduplicated via `sort -u`) tracked. All 19 tracked job IDs individually verified present (`expect-job` loop all `ok`); the 20th job's identity was not captured (cluster had already been torn down by the script's EXIT trap by the time this was investigated). This is the OPPOSITE of bug-661/795's symptom (loss) \u2014 an unexplained EXTRA job, not a missing one. 4 of the other 5 runs (and every prior round within run 5 itself, rounds 1-8) showed correct exact-count durability with zero anomalies.",
|
||||
"file": "src/tools/test-commit-durability.sh (harness) or controller job-recovery path (unconfirmed which)",
|
||||
"root_cause": "NOT INVESTIGATED \u2014 deprioritized because it doesn't match the loss scenario task 6.1 was probing for, and reproducing it requires re-running with a 'joblist' capture at the exact expect-jobs failure point (the script doesn't dump the full list on a count mismatch, only the count). Two live hypotheses, unconfirmed: (a) harness-side race \u2014 ACKED_FILE's write (submit_n) and run_cold_restart_recovery's read (`committed=$(sort -u ACKED_FILE | grep -c .)`) are not synchronized against a submission whose ack arrives asynchronously after the harness moves on, so a genuinely-committed job simply never got recorded in ACKED_FILE \u2014 a test-harness bookkeeping gap, not a controller bug; (b) a genuine job-duplication bug on the controller's recovery/replay path creating an extra job entry with a fresh ID. Also observed in this run's ctrl-1.log: repeated unrelated `handle_jobsubmit fail to lock job 63-15--1-1015808[32751]` errors and one `job not durable (rc=-110) \u2014 rejecting submission` for a DIFFERENTLY-FORMATTED job ID (not this harness's `0-0-<timestamp>-0` convention) \u2014 almost certainly a stray unrelated process hitting the same cluster ports in the shared sandbox, not this test's own traffic, but flagged here in case it's connected.",
|
||||
"fix": "NOT YET ROOT-CAUSED (still needs a live re-run to capture the actual extra job's identity). Made the harness able to diagnose it next time it fires: src/tools/test-commit-durability.sh's assert_job_count() now dumps the live `joblist` output AND the ACKED_FILE ledger (sorted, deduplicated) on any count mismatch, instead of only printing the count as before. This was the bug's own recorded 'next step'. Bash syntax-checked (bash -n); not yet re-run against a live cluster in this session (that requires reproducing the original 5-round real-fs soak, which is long-running).",
|
||||
"fix": "STILL NOT ROOT-CAUSED for the original exact symptom (successful query, count=20 vs expected=19) \u2014 2 fresh repro attempts (2026-07-09, 3 rounds each, real ext4 fs under ~/tmp) did not reproduce an extra job; both runs showed the tracked count exactly matched after the fix below. HOWEVER, both attempts DID hit a DIFFERENT, related bug in the exact same assert_job_count() check: right after the cold-restart-all election, the aggregate expect-jobs/joblist query itself FAILED outright with 'xce_joblist: Unknown error 10014' / 'could not determine job count' (a query FAILURE, not a wrong-but-successful count) \u2014 this is the identical client-side race as bug-1422 (libxce's failover retry loop, client_failover.c, can exhaust its 20-attempt budget in a rapid post-election retry storm and surface a stale E_XCE_STANDBY). Confirmed as a false alarm both times: the very next per-id assert_no_loss check (individual expect-job calls) found every job present. Fixed assert_job_count() with the same bounded-retry pattern used for bug-1422 (3 attempts, 2s settle) \u2014 re-run after the fix: 7 passed, 0 failed, clean. This closes the QUERY-FAILURE false-negative class in this check, and makes the harness reliable enough to actually catch the ORIGINAL extra-job symptom if it recurs (previously a query failure and a genuine extra-job count were indistinguishable at a glance; now the retry-exhausted case self-heals and no longer masks a real mismatch). The original 'extra job' phenomenon itself remains open/unconfirmed \u2014 needs a longer run (original was 1-of-5 full 8-round soaks) to have a realistic chance of firing again.",
|
||||
"tags": [
|
||||
"ha-commit-durability",
|
||||
"low-priority",
|
||||
"cold-restart-all",
|
||||
"job-count-mismatch",
|
||||
"harness-instrumented",
|
||||
"unconfirmed"
|
||||
"unconfirmed",
|
||||
"related-fix-applied"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-661"
|
||||
"bug-661",
|
||||
"bug-1422"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-02T16:00:00.000Z"
|
||||
"occurrences": 3,
|
||||
"last_seen": "2026-07-09T10:45:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1397",
|
||||
|
|
@ -12330,8 +12332,8 @@
|
|||
"timestamp": "2026-07-03T08:00:00.000Z",
|
||||
"error_message": "controller_unit_test SIGSEGV (full run, no --run filter): Thread 1 crashes in SSL_CTX_free() called from iomgr_fini() during test_ha_vep_teardown.c:271 (test_vep_teardown_runs_after_inflight), which runs at controller_test.c main() TEST(test_ha_vep_teardown). Kills the whole binary before any later-registered TEST() can run, so a bare `controller_unit_test` invocation with no --run filter never reaches a final summary and the exit code is 139 (SIGSEGV), not a Unity pass/fail count.",
|
||||
"file": "src/controller/testsuite/test_ha_vep_teardown.c",
|
||||
"root_cause": "Not yet root-caused. Reproduced deterministically (2/2 tries) on raft_tla @ 3ecb724 (BEFORE any raft-apply-decoupling task-1.1 commit.c/commit.h changes), so it predates and is unrelated to that feature. gdb backtrace: SSL_CTX_free (libssl.so.3) <- iomgr_fini (netio.c:1262) <- test_vep_teardown_runs_after_inflight (test_ha_vep_teardown.c:271) <- UnityDefaultTestRun <- test_ha_vep_teardown (test_ha_vep_teardown.c:630) <- main (controller_test.c). Likely an SSL_CTX double-free/use-after-free or teardown-ordering bug in iomgr_fini, not in commit.c.",
|
||||
"fix": "NOT FIXED (out of scope for raft-apply-decoupling task 1.1 -- discovered incidentally while independently verifying that task's regression-safety claim; task 1.1's boundary is commit.c/commit.h only and does not touch netio.c/iomgr). Workaround used for task 1.1's own regression check: run controller_unit_test with an explicit --run allowlist that excludes test_ha_vep_teardown (or run test groups individually), rather than the bare no-arg invocation.",
|
||||
"root_cause": "ROOT-CAUSED 2026-07-09: iomgr_init() (src/lib/netio.c:1168-1184) only ever initialized iomgr_t's `lock` and `epoll_fd` fields -- it never set `ssl_ctx` (or `worker`). Every production caller embeds iomgr_t inside a calloc'd/static context struct (cluster_t.iomgr, xce_ctx_t.iomgr), which masked the gap. test_ha_vep_teardown.c stack-allocates `iomgr_t iomgr;` at 4 call sites (lines 235, 294, 349, 476) WITHOUT a memset -- unlike the sibling libxce tests (test_netio_arm_failure.c:52, test_request_send_propagation.c:34) which explicitly `memset(&iomgr, 0, sizeof(iomgr))` before iomgr_init, establishing the real (undocumented) contract. Left un-zeroed, `ssl_ctx` holds whatever garbage bit pattern occupies that stack slot -- non-deterministic across runs/process history (confirmed intermittent: some full-suite runs exit 0, others SIGSEGV, matching the reported non-strictly-deterministic behavior). iomgr_fini()'s `if (iomgr->ssl_ctx != NULL) SSL_CTX_free(...)` guard then passes on that garbage pointer, and SSL_CTX_free() dereferences a wild address -> SIGSEGV. Proven deterministically with a standalone repro (poisoned an iomgr_t with 0xAB, called the real iomgr_init()+iomgr_fini() from libxce.a) producing an identical backtrace (SSL_CTX_free <- iomgr_fini netio.c:1262 <- caller). gdb backtrace from the original report: SSL_CTX_free (libssl.so.3) <- iomgr_fini (netio.c:1262) <- test_vep_teardown_runs_after_inflight (test_ha_vep_teardown.c:271) <- UnityDefaultTestRun <- test_ha_vep_teardown (test_ha_vep_teardown.c:630) <- main (controller_test.c). Not a double-free/UAF and not a cross-test state leak -- iomgr_set_ssl_ctx() is never called anywhere in test_ha_vep_teardown.c, so no real SSL_CTX ownership is even in play; it is purely uninitialized-memory read.",
|
||||
"fix": "FIXED 2026-07-09: src/lib/netio.c iomgr_init() now explicitly sets `iomgr->ssl_ctx = NULL;` and `iomgr->worker = 0;` before touching lock/epoll_fd, so iomgr_init() fully establishes iomgr_t's invariants itself instead of assuming the caller pre-zeroed the struct -- mirrors the same file's do_endpoint_init(), which already fully initializes every endpoint_t field rather than relying on calloc. Verified safe for all production callers (xcectx.c, myapp.c, cluster.c): every one calls iomgr_set_ssl_ctx() AFTER iomgr_init(), never before, so clearing ssl_ctx in iomgr_init cannot clobber a caller-supplied value. No test file was modified -- the library-level fix protects every current and future caller, including the 4 unguarded stack declarations in test_ha_vep_teardown.c. Verified: (1) standalone ASAN repro poisoning the stack before iomgr_init now shows ssl_ctx == NULL post-init and iomgr_fini() returns cleanly instead of crashing; (2) rebuilt controller_unit_test and ran the full suite (no --run filter) repeatedly under ASAN (detect_leaks=0) with zero SIGSEGVs across all runs -- one full run captured 265 Tests / 0 Failures / 0 Ignored, with all 6 tests in the test_ha_vep_teardown group (including test_vep_teardown_runs_after_inflight) passing their own assertions, not just failing to crash.",
|
||||
"tags": [
|
||||
"testsuite",
|
||||
"controller",
|
||||
|
|
@ -12339,11 +12341,12 @@
|
|||
"ssl",
|
||||
"iomgr",
|
||||
"pre-existing",
|
||||
"raft-apply-decoupling"
|
||||
"raft-apply-decoupling",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-07-03T08:00:00.000Z"
|
||||
"last_seen": "2026-07-09T12:45:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1407",
|
||||
|
|
@ -12688,10 +12691,10 @@
|
|||
{
|
||||
"id": "bug-1418",
|
||||
"timestamp": "2026-07-08T03:20:00.000Z",
|
||||
"error_message": "[xce-test-ha --chaotic 10-round fresh binary] Rounds 1-4 clean, round 5+ permanent liveness deadlock. Both ctrl-2 and ctrl-3 win raft elections (became_leader) but promotion REFUSED every ~12s: apply_divergent=1 durable_floor=26 cluster_hwm=26 peer_reachable=1. Terms escalate 31→47+, no primary emerges for rounds 5-10.",
|
||||
"error_message": "[xce-test-ha --chaotic 10-round fresh binary] Rounds 1-4 clean, round 5+ permanent liveness deadlock. Both ctrl-2 and ctrl-3 win raft elections (became_leader) but promotion REFUSED every ~12s: apply_divergent=1 durable_floor=26 cluster_hwm=26 peer_reachable=1. Terms escalate 31\u219247+, no primary emerges for rounds 5-10.",
|
||||
"file": "src/controller/ha.c (raft_cb_become_leader promotion gate)",
|
||||
"root_cause": "RESIDUAL after bug-1418 layer-1 (fe4352f WAL dup-submit) and layer-2 (0eb1054 checkpoint-lsn) fixes — both layers confirmed working (sequential 10-round PASS, zero job loss, all 3 nodes led). The promotion gate (raft-apply-decoupling Req 8.4) correctly detects APPLY_DIVERGENT, but has NO recovery path when durable_floor >= cluster_hwm. In this case: durable_floor(26) == cluster_hwm(26) — node HAS the full committed WAL log, has NOT yet applied it (apply pump lagged behind WAL tip during kill). The gate requires applied_lsn >= cluster_hwm to promote, but no primary exists to stream entries for apply catch-up → circular deadlock. The node COULD self-apply (its WAL contains all committed entries) but the gate won't let it become primary to do so. Sequential mode avoids this (waits for full convergence → apply catches up before next kill); chaotic mode triggers it reliably (restarts a node then immediately kills the primary, so the restarted node's applied_lsn lags). Unlike prior bug-1418 occurrences, this is NOT a mislabeled watermark — it's a genuine applied-vs-committed gap that the gate correctly identifies but has no mechanism to resolve.",
|
||||
"fix": "NOT FIXED. Proposed: relax promotion gate to allow CTRL_PRIMARY when durable_floor >= cluster_hwm (node has full committed log) even if apply_divergent=1. The new primary self-applies committed entries (apply_committed_entry is idempotent per Req 8.1) before serving read traffic. The gate should only permanently fence when durable_floor < cluster_hwm (node physically missing committed entries — would need snapshot backfill).",
|
||||
"root_cause": "RESIDUAL after bug-1418 layer-1 (fe4352f WAL dup-submit) and layer-2 (0eb1054 checkpoint-lsn) fixes \u2014 both layers confirmed working (sequential 10-round PASS, zero job loss, all 3 nodes led). The promotion gate (raft-apply-decoupling Req 8.4) correctly detects APPLY_DIVERGENT, but has NO recovery path when durable_floor >= cluster_hwm. In this case: durable_floor(26) == cluster_hwm(26) \u2014 node HAS the full committed WAL log, has NOT yet applied it (apply pump lagged behind WAL tip during kill). The gate requires applied_lsn >= cluster_hwm to promote, but no primary exists to stream entries for apply catch-up \u2192 circular deadlock. The node COULD self-apply (its WAL contains all committed entries) but the gate won't let it become primary to do so. Sequential mode avoids this (waits for full convergence \u2192 apply catches up before next kill); chaotic mode triggers it reliably (restarts a node then immediately kills the primary, so the restarted node's applied_lsn lags). Unlike prior bug-1418 occurrences, this is NOT a mislabeled watermark \u2014 it's a genuine applied-vs-committed gap that the gate correctly identifies but has no mechanism to resolve.",
|
||||
"fix": "NOT FIXED. Proposed: relax promotion gate to allow CTRL_PRIMARY when durable_floor >= cluster_hwm (node has full committed log) even if apply_divergent=1. The new primary self-applies committed entries (apply_committed_entry is idempotent per Req 8.1) before serving read traffic. The gate should only permanently fence when durable_floor < cluster_hwm (node physically missing committed entries \u2014 would need snapshot backfill).",
|
||||
"tags": [
|
||||
"ha",
|
||||
"raft",
|
||||
|
|
@ -12703,7 +12706,11 @@
|
|||
"residual",
|
||||
"self-apply-recovery"
|
||||
],
|
||||
"related_bugs": ["bug-1418", "bug-1419", "bug-1417"],
|
||||
"related_bugs": [
|
||||
"bug-1418",
|
||||
"bug-1419",
|
||||
"bug-1417"
|
||||
],
|
||||
"occurrences": 4,
|
||||
"last_seen": "2026-07-08T03:20:00.000Z"
|
||||
},
|
||||
|
|
@ -12712,7 +12719,7 @@
|
|||
"timestamp": "2026-07-08T06:55:15.000Z",
|
||||
"error_message": "Sequential HA soak regression: no primary elected at term>8 by round 6 (initially suspected to stem from the bug-1417 fix); root-caused 2026-07-09 as the primary never discovering a follower's higher term via AppendEntries-reject responses",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "ha_raft_on_repl_ack only routes AE-reply term-adoption for success=1 ACKs; AE rejects (success=0) never reach raft_on_append_entries_reply. That function additionally drops any reply with reply_term != s->term (raft.c:832-833, stale-term drop) before it can adopt a higher term. Net effect: once a follower is ahead in term (e.g. it steps down from PreVote to CTRL_STANDBY at a higher term while an existing primary keeps campaigning/replicating at its old term), every AppendEntries the stale primary sends is rejected with ae_reject_stale, the primary never learns the higher term, and it can never regain write-quorum (needs 2/3, only self-vote counts) — a permanent livelock. First surfaced as an election regression (sequential soak: no primary elected at term>8 by round 6, chaotic stalled at round 5) suspected to be caused by the bug-1417 vote-index-mapping fix; re-reproduced 2026-07-09 as a round-4 sequential stall (primary ctrl-2 term=6, follower ctrl-3 term=7, 490 ae_reject_stale events, primary writable_lsn stuck at 17), which pinned the exact mechanism above.",
|
||||
"root_cause": "ha_raft_on_repl_ack only routes AE-reply term-adoption for success=1 ACKs; AE rejects (success=0) never reach raft_on_append_entries_reply. That function additionally drops any reply with reply_term != s->term (raft.c:832-833, stale-term drop) before it can adopt a higher term. Net effect: once a follower is ahead in term (e.g. it steps down from PreVote to CTRL_STANDBY at a higher term while an existing primary keeps campaigning/replicating at its old term), every AppendEntries the stale primary sends is rejected with ae_reject_stale, the primary never learns the higher term, and it can never regain write-quorum (needs 2/3, only self-vote counts) \u2014 a permanent livelock. First surfaced as an election regression (sequential soak: no primary elected at term>8 by round 6, chaotic stalled at round 5) suspected to be caused by the bug-1417 vote-index-mapping fix; re-reproduced 2026-07-09 as a round-4 sequential stall (primary ctrl-2 term=6, follower ctrl-3 term=7, 490 ae_reject_stale events, primary writable_lsn stuck at 17), which pinned the exact mechanism above.",
|
||||
"fix": "Added two fallback term-discovery vectors that fire regardless of AE-reject outcome. (1) StateReport heartbeat-reply now carries the standby's current term (controller.capnp +term field @2, typemap.h state_report_t +term, ha_nop_reply_state_report populates it from ha->term) so the primary learns a higher term on every convergence cycle. (2) New library entry point raft_on_higher_term_observed(s, observed_term) mirrors UpdateTerm (strictly-greater-than only; touches term/voted_for/role; on_persist before on_become_follower for durability ordering) and is called from repl_ack_decode_and_dispatch on both the heartbeat state-report branch and the WAL-NACK branch (ha.c). Primary now steps down instead of retrying AppendEntries against a follower that rejects forever. Validated: 265 controller + 287 lib unit tests pass (zero regressions); 10-round sequential soak PASS (33 jobs, zero loss, all 3 nodes led ctrl-1/ctrl-2/ctrl-3); 10-round chaotic soak PASS (33 jobs, zero committed divergence, zero committed job loss).",
|
||||
"tags": [
|
||||
"ha",
|
||||
|
|
@ -12724,17 +12731,21 @@
|
|||
"replication",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": ["bug-1417", "bug-1418", "bug-851"],
|
||||
"related_bugs": [
|
||||
"bug-1417",
|
||||
"bug-1418",
|
||||
"bug-851"
|
||||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-07-09T08:50:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1422",
|
||||
"timestamp": "2026-07-09T10:14:14.000Z",
|
||||
"error_message": "40-round sequential HA soak FAIL at round 36: verify_no_loss reported '0/111 jobs confirmed present' right after a fresh election (new primary ctrl-1, term=40) — xce_test's expect-job got E_XCE_STANDBY (10014) from a fresh client session whose own wait-primary had JUST confirmed ctrl-1 as serving 45ms earlier",
|
||||
"error_message": "40-round sequential HA soak FAIL at round 36: verify_no_loss reported '0/111 jobs confirmed present' right after a fresh election (new primary ctrl-1, term=40) \u2014 xce_test's expect-job got E_XCE_STANDBY (10014) from a fresh client session whose own wait-primary had JUST confirmed ctrl-1 as serving 45ms earlier",
|
||||
"file": ".claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"root_cause": "False-positive test-harness failure, NOT real committed-job loss: ctrl-1's own controller log confirms it stayed CTRL_PRIMARY throughout the window and successfully committed the 3 new post-failover jobs (log_append/ae_reply_success/commit_advanced at 10:14:22-23). ha_dispatch_gate's CTRL_PRIMARY branch (ha.c:3451-3531) unconditionally permits OPCode_get (joblist) even while fenced, so ctrl-1 itself could not have returned E_XCE_STANDBY for this query — the reply must have come from a standby (ctrl-2, just killed, or ctrl-3) that the client's failover retry loop landed on. Root cause traced to client_failover.c's xcecli_retry_next/xcecli_retry_decide (client_failover.c:305-319): E_XCE_STANDBY classifies as XCE_FO_RETRY_NOW (client_failover.c:59-60, prompt rotate, no backoff), and the 20-attempt retry budget (3 masters) should reliably reach the real primary — but when the WHOLE budget is exhausted while every recent classification was RETRY_NOW (as opposed to RETRY_BACKOFF), xcecli_retry_decide's budget-exhaustion branch (line 312-318) does NOT override *rc the way it does for RETRY_BACKOFF exhaustion (which maps to the distinct -E_XCE_NOT_READY, line 316-317) — it just returns false with *rc left at whatever the last attempt's raw status was (here, a stray E_XCE_STANDBY from a standby hit during the rapid post-election retry storm). verify_no_loss() (run-ha-soak.sh) then did a single one-shot check with zero retry at the driver level, unlike the job-submit loop just above it in the same file which already retries on this exact 'primary likely just promoted, regaining write-quorum' condition (run-ha-soak.sh:483-486). Only surfaced at round 36 of a 40-round run — narrow enough timing that shorter 6-10 round soaks across many prior sessions never hit it.",
|
||||
"fix": "Added a bounded retry (3 total attempts, 2s settle sleep between) around the WHOLE verify_no_loss() check in run-ha-soak.sh, mirroring the existing submit-retry loop's rationale: a job still missing after a real settle window + 2 retries is a genuine loss, not this client-side retry-storm race. Did not touch client_failover.c (the battle-tested libxce-failover retry/rotation library with many other production consumers) — fixing at the test-driver level is safer and fully local. Re-ran both 40-round soaks after the fix: sequential PASS (123 jobs, zero loss, all 3 nodes led) and chaotic PASS (123 jobs, zero committed divergence, zero committed job loss).",
|
||||
"root_cause": "False-positive test-harness failure, NOT real committed-job loss: ctrl-1's own controller log confirms it stayed CTRL_PRIMARY throughout the window and successfully committed the 3 new post-failover jobs (log_append/ae_reply_success/commit_advanced at 10:14:22-23). ha_dispatch_gate's CTRL_PRIMARY branch (ha.c:3451-3531) unconditionally permits OPCode_get (joblist) even while fenced, so ctrl-1 itself could not have returned E_XCE_STANDBY for this query \u2014 the reply must have come from a standby (ctrl-2, just killed, or ctrl-3) that the client's failover retry loop landed on. Root cause traced to client_failover.c's xcecli_retry_next/xcecli_retry_decide (client_failover.c:305-319): E_XCE_STANDBY classifies as XCE_FO_RETRY_NOW (client_failover.c:59-60, prompt rotate, no backoff), and the 20-attempt retry budget (3 masters) should reliably reach the real primary \u2014 but when the WHOLE budget is exhausted while every recent classification was RETRY_NOW (as opposed to RETRY_BACKOFF), xcecli_retry_decide's budget-exhaustion branch (line 312-318) does NOT override *rc the way it does for RETRY_BACKOFF exhaustion (which maps to the distinct -E_XCE_NOT_READY, line 316-317) \u2014 it just returns false with *rc left at whatever the last attempt's raw status was (here, a stray E_XCE_STANDBY from a standby hit during the rapid post-election retry storm). verify_no_loss() (run-ha-soak.sh) then did a single one-shot check with zero retry at the driver level, unlike the job-submit loop just above it in the same file which already retries on this exact 'primary likely just promoted, regaining write-quorum' condition (run-ha-soak.sh:483-486). Only surfaced at round 36 of a 40-round run \u2014 narrow enough timing that shorter 6-10 round soaks across many prior sessions never hit it.",
|
||||
"fix": "Added a bounded retry (3 total attempts, 2s settle sleep between) around the WHOLE verify_no_loss() check in run-ha-soak.sh, mirroring the existing submit-retry loop's rationale: a job still missing after a real settle window + 2 retries is a genuine loss, not this client-side retry-storm race. Did not touch client_failover.c (the battle-tested libxce-failover retry/rotation library with many other production consumers) \u2014 fixing at the test-driver level is safer and fully local. Re-ran both 40-round soaks after the fix: sequential PASS (123 jobs, zero loss, all 3 nodes led) and chaotic PASS (123 jobs, zero committed divergence, zero committed job loss).",
|
||||
"tags": [
|
||||
"xce-test-ha",
|
||||
"false-positive",
|
||||
|
|
@ -12744,8 +12755,299 @@
|
|||
"test-harness",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": ["bug-854", "bug-1421"],
|
||||
"related_bugs": [
|
||||
"bug-854",
|
||||
"bug-1421"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-09T10:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1418",
|
||||
"timestamp": "2026-07-09T10:30:00.000Z",
|
||||
"error_message": "Final resolution pass over bug-1418 (all 4 prior occurrences): the two contributing factors from the 2026-07-07T18:20 analysis (term desync + apply-divergence promotion-gate liveness deadlock) are both now resolved, not by the originally-proposed gate relaxation.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "Re-audited both factors against the CURRENT code (post raft-apply-decoupling 32/32, post bug-1421): (1) TERM DESYNC \u2014 ha_start_prevote_async (ha.c:6715) already computes its prospective term from ha->raft.term + 1, not the stale ha->term the original analysis described; ha.c:1009-1010 explicitly syncs ha->term from the raft term on every relevant transition. This half was already fixed independently of bug-1418's own tracked fix. (2) APPLY-DIVERGENCE LIVENESS DEADLOCK \u2014 the promotion gate (ha.c raft_cb_become_leader ~line 1130, and promotion_fence_timeout_fn ~line 876) unconditionally refuses promotion while persist_apply_verdict()==APPLY_DIVERGENT, by deliberate design (raft-apply-decoupling task 5.3, Req 8.4: a divergent local materialization must never be trusted to self-clear via any live/structural condition, including durable_floor>=cluster_hwm \u2014 only a fresh persist_recover() replay may clear it). The buglog's own previously-proposed fix (relax the gate to self-apply when durable_floor>=cluster_hwm) would have violated this intentional invariant and risked serving genuinely divergent state to clients. Instead, raft-apply-decoupling's tasks 5.1 (checkpoint_lsn mislabeled from WAL tip instead of the logical applied cursor, causing a false APPLY_DIVERGENT on an ordinary crash-during-checkpoint) and 7.2/bug-1416 (a boot-time TOCTOU where a returning node's persist_recover() replay-verify raced the live apply pump / live-replication apply path, producing a false APPLY_DIVERGENT on the restart/catch-up path specifically \u2014 the exact scenario bug-1418's residual occurrence hit: 'apply pump lagged behind WAL tip during kill') closed the false-positive detection paths that were producing spurious APPLY_DIVERGENT verdicts in the first place. A genuine (non-false-positive) divergence still correctly fences forever by design.",
|
||||
"fix": "No code change to the promotion gate itself \u2014 it is intentionally correct and must stay unconditional. The actual fix is the combination of already-shipped upstream work: bug-1421 (AE-reject term-discovery blindspot, this session) + raft-apply-decoupling tasks 5.1/5.3/7.2 (false-APPLY_DIVERGENT elimination, closed prior to this session). Validated: two 40-round soaks today (sequential 123 jobs zero-loss all-3-nodes-led; chaotic 123 jobs zero-committed-divergence zero-job-loss) \u2014 an order of magnitude more election churn than the original 10-round soak that first surfaced bug-1418 \u2014 produced zero apply-divergent promotion refusals and zero liveness deadlocks. Further corroborated by the 30-round sequential+chaotic regression run immediately after this entry.",
|
||||
"tags": [
|
||||
"ha",
|
||||
"raft",
|
||||
"apply-divergence",
|
||||
"promotion-gate",
|
||||
"chaotic-failover",
|
||||
"liveness",
|
||||
"term-desync",
|
||||
"resolved-via-upstream",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1417",
|
||||
"bug-1419",
|
||||
"bug-1421",
|
||||
"bug-1416"
|
||||
],
|
||||
"occurrences": 5,
|
||||
"last_seen": "2026-07-09T10:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1423",
|
||||
"timestamp": "2026-07-09T13:58:16.000Z",
|
||||
"error_message": "30-round sequential HA soak FAIL at round 5: post-failover submit did not get 3 acks after 5 retry attempts (15s+). Root-caused as a genuine permanent election deadlock, not a client-side false positive: ctrl-3 won PreVote and entered CTRL_ELECTING (real election, term=10) with ctrl-1 dead and ctrl-2 alive, then never showed another HA state-machine event for 2+ minutes (until the harness killed the cluster on FAIL) \u2014 stuck in CTRL_ELECTING forever. Meanwhile ctrl-2 endlessly retried its own PreVote every ~15-17s, correctly denied every time by ctrl-3 on freshness grounds, but that denial never unstuck ctrl-3's already-in-flight real election.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "raft_cb_send_vote_request (ha.c, sends the real RequestVote RPC to each peer) has two failure branches \u2014 endpoint_connect() failing and iomgr_monitor() failing \u2014 that are exactly what happens when the RPC target is a dead/unreachable peer (here, ctrl-1, killed at the start of round 5). The endpoint_connect failure branch only called the legacy handle_request_vote_reply() when `c == NULL` (unit-test mode) \u2014 in PRODUCTION (c always non-NULL) a connect failure to a dead peer was silently dropped with NO notification to the raft library at all. The iomgr_monitor failure branch called handle_request_vote_reply() unconditionally, including in production \u2014 but per vote_reply_dispatch's own design comment, handle_request_vote_reply is the LEGACY/ctx-done-signaling path; 'raft_on_vote_reply remains the SOLE production promotion driver' (vote_reply_dispatch calls ha_record_vote_reply + raft_on_vote_reply, NOT handle_request_vote_reply). Net effect: a candidate's vote request to a dead peer never registers as a 'reply' with the production raft library's raft_state_t.replies_received counter. raft.c's Step 6 fast-path ('all peers have replied, no quorum \u2192 become follower and retry', gated on replies_received == n_peers-1) can therefore NEVER trigger whenever any configured peer is down \u2014 with quorum=2 in a 3-node cluster and one peer dead, replies_received caps at 1 (only the live peer's real reply, if any) and can never reach n_peers-1=2. Recovery then depends ENTIRELY on the raw per-campaign election_timeout (5-10s, ha_jittered_election_delay) firing via delaymgr \u2014 in the observed failure, no election_timeout_fn trace appeared in the log at all for the full 2+ minute window, leaving the candidate permanently wedged.",
|
||||
"fix": "Routed both raft_cb_send_vote_request failure branches (connect failure and monitor failure) through vote_reply_dispatch(ha, ectx, votes_idx, term, vote_granted=0, 0, 0) in production (c != NULL) \u2014 the exact same call a genuine vote reply makes (ha_record_vote_reply + raft_on_vote_reply), so a dead/unreachable peer now correctly counts as 'replied, not granted' immediately on connect/monitor failure instead of only via the timeout. Test mode (c == NULL) keeps the legacy handle_request_vote_reply call unit tests depend on. Added a forward declaration of the previously-file-local-only vote_reply_dispatch near the top of ha.c so raft_cb_send_vote_request (defined much earlier in the file) can call it. Verified: full controller_unit_test suite clean (755 PASS / 0 Fail, exit 0) both before and after removing temporary diagnostic logging used during root-cause; re-ran the exact failing scenario \u2014 30-round sequential HA soak PASS (93 jobs, zero loss, all 3 nodes led) and 30-round chaotic HA soak PASS (93 jobs, zero committed divergence, zero committed job loss) \u2014 both previously hit this stall reliably (round 5 of the sequential run, 100% reproduction before the fix).",
|
||||
"tags": [
|
||||
"ha",
|
||||
"raft",
|
||||
"election",
|
||||
"liveness-deadlock",
|
||||
"vote-request",
|
||||
"dead-peer",
|
||||
"quorum",
|
||||
"permanent-stall",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1417",
|
||||
"bug-1418",
|
||||
"bug-1421"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-09T14:40:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1424",
|
||||
"timestamp": "2026-07-09T17:00:00.000Z",
|
||||
"error_message": "PreVote twin of bug-1423: raft_cb_send_pre_vote_request's connect/monitor failure branches (a dead/unreachable peer) never registered a reply with the raft library's PreVote tally, in EITHER test or production mode (worse than the real-vote path, which at least handled test mode).",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "Same missing-registration pattern as bug-1423, in raft_cb_send_pre_vote_request instead of raft_cb_send_vote_request. Found while auditing for the analogous gap after fixing bug-1423. Lower severity than bug-1423: PreVote has an unconditional round-timeout fallback (ha_prevote_round_timeout_fn, confirmed firing reliably every round across dozens of observed rounds in the bug-1423 investigation logs) that always resolves the round regardless of tally completeness, so this was never a permanent-deadlock class bug like bug-1423 was \u2014 just a latency waste: every PreVote round against a down peer burns the full round timeout instead of concluding as soon as the live peer(s) reply.",
|
||||
"fix": "Routed both failure branches through pre_vote_reply_dispatch(ha, c, peer_idx, granted=0) \u2014 the same production PreVote-reply path a real reply uses (raft_on_pre_vote_reply) \u2014 mirroring bug-1423's fix exactly. Added a forward declaration near the top of ha.c (pre_vote_reply_dispatch is defined much later in the file). Verified: full controller_unit_test suite clean (0 Fail, exit 0) after the change.",
|
||||
"tags": [
|
||||
"ha",
|
||||
"raft",
|
||||
"prevote",
|
||||
"dead-peer",
|
||||
"latency",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1423"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-09T17:05:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1425",
|
||||
"timestamp": "2026-07-09T21:15:00.000Z",
|
||||
"error_message": "Intermittent (~2 of 9 attempts) client hang in the synchronous jobsub (COMMIT_MODE_WAIT) path shortly after a fresh single-node election.",
|
||||
"file": "src/controller/persist.c or src/controller/commit.c (unrooted)",
|
||||
"root_cause": "NOT ROOT-CAUSED. Flagged during raft-apply-decoupling task 7.4's throughput benchmark work. Reproduced identically on both the pre-fix and post-fix (task 6.1 apply-pump) binaries \u2014 confirmed unrelated to that fix. Resembles but is not confirmed identical to the closed bug-1416 boot-race class (persist_recover() replay-verify racing the live apply pump / live-replication apply path).",
|
||||
"fix": "NOT FIXED. Worked around via retry for task 7.4's benchmarking purposes only. Needs a dedicated root-cause investigation with the same live-repro rigor bug-1416 got (gdb-attach via GDB_WRAP seam, or targeted debug logging around the commit-wait barrier right after a fresh single-node election).",
|
||||
"tags": [
|
||||
"ha",
|
||||
"commit-engine",
|
||||
"election",
|
||||
"intermittent",
|
||||
"unconfirmed",
|
||||
"raft-apply-decoupling-followup"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1416"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-09T21:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1426",
|
||||
"timestamp": "2026-07-09T21:15:00.000Z",
|
||||
"error_message": "trim_wal_segments's retention floor has no awareness of the local apply pump's own lag \u2014 a reachable scenario since apply-lag-behind-commit is raft-apply-decoupling's entire premise.",
|
||||
"file": "src/controller/persist.c (trim_wal_segments, apply_pump_read_entry cache)",
|
||||
"root_cause": "Identified during raft-apply-decoupling task 7.4's apply-pump segment-read cache fix (round-2 review finding): the new fast-path cache can (POSIX unlink-keeps-fd-alive) silently advance last_applied_lsn past already-deleted data if the pump is lagging when trim_wal_segments fires. Currently bounded by the pre-existing APPLY_DIVERGENT boot fence (deferred loud fencing on next restart, not silent corruption) \u2014 a real behavior change from the pre-cache code, not literally risk-free, but not an active defect either. Same family as an already-tracked standby-cursor variant (per task 7.4's own notes).",
|
||||
"fix": "NOT FIXED \u2014 deliberately deferred. task 7.4's own scope explicitly excluded redesigning trim_wal_segments's retention-floor awareness of apply-pump lag; documented via an accurate code comment instead. A future fix would give trim_wal_segments a floor input derived from the local apply cursor, not just standby-acked LSNs.",
|
||||
"tags": [
|
||||
"ha",
|
||||
"persistence",
|
||||
"wal-trim",
|
||||
"apply-pump",
|
||||
"low-priority",
|
||||
"raft-apply-decoupling-followup"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-09T21:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1427",
|
||||
"timestamp": "2026-07-09T21:15:00.000Z",
|
||||
"error_message": "APPLY_DIVERGENT's entity-count correctness check (persist_verify_apply_correctness) covers job entities only \u2014 worker entities can diverge (worker_from_capnp()/restore_worker() genuinely creates a new worker on a WAL_WORKER_CTRL/HOOK_DETAIL_WORKER_JOIN entry when absent) but this is currently invisible to the check.",
|
||||
"file": "src/controller/persist.c (persist_verify_apply_correctness, count_jobmgr_entries)",
|
||||
"root_cause": "Disclosed, accepted scope gap from raft-apply-decoupling task 5.1's own design: the entity-count check was scoped to jobs only (queues correctly excluded \u2014 config-derived, WAL_QUEUE_CTRL never creates a new one, verified by two review rounds). Workers were never brought in scope. Not a regression \u2014 a known, deliberate boundary, just not yet extended.",
|
||||
"fix": "NOT FIXED. Extend persist_verify_apply_correctness() with a worker-count check mirroring count_jobmgr_entries() if/when a worker-creation divergence needs to be closed (task 5.1's own forward note).",
|
||||
"tags": [
|
||||
"ha",
|
||||
"persistence",
|
||||
"apply-divergent",
|
||||
"worker",
|
||||
"low-priority",
|
||||
"raft-apply-decoupling-followup"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-09T21:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1428",
|
||||
"timestamp": "2026-07-09T21:15:00.000Z",
|
||||
"error_message": "Narrow TOCTOU residual in persist_register_pending_apply() (task 6.2, part of bug-1415's fix) \u2014 not reliably exercised by any current test.",
|
||||
"file": "src/controller/persist.c (persist_register_pending_apply)",
|
||||
"root_cause": "Flagged during raft-apply-decoupling task 6.2 as a non-blocking fast-follow while closing bug-1415. Specific race window not fully documented in tasks.md's own notes; needs a fresh look at the function to characterize exactly what window remains and whether it's reachable in production timing.",
|
||||
"fix": "NOT FIXED \u2014 explicitly deferred as a fast-follow, not a blocker for task 6.2 or the spec's closure. Needs: (1) characterize the exact TOCTOU window, (2) determine production reachability, (3) add a regression test that exercises it before fixing.",
|
||||
"tags": [
|
||||
"ha",
|
||||
"persistence",
|
||||
"apply-pump",
|
||||
"toctou",
|
||||
"unconfirmed",
|
||||
"raft-apply-decoupling-followup"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1415"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-09T21:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1429",
|
||||
"timestamp": "2026-07-10",
|
||||
"error_message": "controller_unit_test suite TIMEOUT (300s, killed by SIGTERM) during /xce-build controller run immediately after uncommitted edits to persist.c, ha_convergence.c, config_repl.c (delaymgr-inline blocking I/O moved to VTASK_LIVE_BLOCK vtasks with inline-fallback-on-failure). Compile succeeded with zero errors (only pre-existing warnings). meson test summary: Ok:0 Fail:0 Timeout:1. Last test output before the hang was test_demotion_replay_yields_committed_only (test_commit_barrier_reply.c) printing overlayfs-SKIP, followed by a burst of HA FSM / persist_apply_pump_run log lines ending in \"HA quorum lost: connected=0 required=1\", then total silence for the remaining ~294s until SIGTERM. No ASAN/crash report -- looks like a genuine hang/deadlock, not a crash.",
|
||||
"file": "src/controller/persist.c, src/controller/ha_convergence.c, src/controller/config_repl.c",
|
||||
"root_cause": "NOT YET ROOT-CAUSED. Suspect the new delaymgr-trampoline -> vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) path blocks waiting for vtask completion that never arrives inside the controller_unit_test harness (e.g. vtask thread pool not pumped/started for this test fixture, or a lock-ordering deadlock between the delaymgr thread and the vtask worker). Ruled out: disk space is not exhausted (162G free, 81% used) so this is NOT the bug-924 ENOSPC/overlayfs-fdatasync pattern despite ~8182 leftover /tmp/xce_* dirs being present.",
|
||||
"fix": "NOT FIXED -- reported to user for diagnosis; no code changes made in this session.",
|
||||
"tags": [
|
||||
"vtask",
|
||||
"VTASK_LIVE_BLOCK",
|
||||
"delaymgr",
|
||||
"timeout",
|
||||
"hang",
|
||||
"deadlock-suspected",
|
||||
"controller-tests",
|
||||
"persist.c",
|
||||
"ha_convergence.c",
|
||||
"config_repl.c"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-924"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-10"
|
||||
},
|
||||
{
|
||||
"id": "bug-1430",
|
||||
"timestamp": "2026-07-10T07:12:00Z",
|
||||
"error_message": "AddressSanitizer: heap-use-after-free on address ... WRITE of size 4 ... #0 ha_prevote_cancel_round_timeout ../controller/ha.c:6712 #1 ha_start_prevote_async ../controller/ha.c:6889 #2 ha_failover_stagger_fn ../controller/ha.c:7259 -- crashed 2 of 3 nodes during a live chaotic HA soak (10-round overlapping-kill mode)",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "ha_prevote_round_timeout_fn (the delaymgr task body for a PreVote round's timeout) only cleared the static s_prevote_round_ctx pointer inside its 'round still in progress' branch (ha->raft.pre_vote_in_progress true). When a round instead got decided via a reply arriving through raft_on_pre_vote_reply (which clears pre_vote_in_progress without touching s_prevote_round_ctx) and the round-timeout task fired afterward anyway (a benign, expected race), the function fell straight through to free(ctx) with s_prevote_round_ctx still pointing at that about-to-be-freed memory. The next PreVote round's ha_start_prevote_async defensively calls ha_prevote_cancel_round_timeout() before arming its own timer, reading the dangling pointer and writing to old->cancelled -- UAF. Same call stack (single delaymgr thread T8) on both crashing nodes, all-100%-reproducible under chaotic-mode's rapid overlapping restarts.",
|
||||
"fix": "ha_prevote_round_timeout_fn now does a CAS (atomic_compare_exchange_strong_explicit matching s_prevote_round_ctx against this specific ctx, clearing to NULL) UNCONDITIONALLY as the first action, before either branch -- so the static pointer is never left dangling regardless of which path is taken. Brings it in line with the already-correct pattern used by the sibling function raft_election_timeout_fn (same file), which unconditionally clears its own stored ctx pointer before checking cancelled.",
|
||||
"tags": [
|
||||
"asan",
|
||||
"use-after-free",
|
||||
"prevote",
|
||||
"delaymgr",
|
||||
"chaotic-soak",
|
||||
"raft"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1423",
|
||||
"bug-1424"
|
||||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-07-10T07:12:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1431",
|
||||
"timestamp": "2026-07-10T10:02:00Z",
|
||||
"error_message": "chaotic HA soak: committed_job_loss=5-8 across 10 rounds (VERDICT result=FAIL) after raft-apply-decoupling task 16 gated standby apply strictly on commit; committed_divergence stayed 0 and all jobs eventually reappeared at final settlement -- a visibility-latency gap, not true data loss",
|
||||
"file": "src/controller/ha.c, src/controller/persist.c, src/include/persist.h",
|
||||
"root_cause": "Two stacked gaps in raft_cb_become_leader promotion, both masked pre-task-16 by eager apply-on-WAL-receipt: (1) persist_truncate_wal_above only sets p->committed_lsn on branches of truncate_wal_above_locked that perform a REAL truncation; its common no-op early-return (leader own log already covers safe_lsn) left committed_lsn stuck at its pre-election value, so newly-computed safe_committed_lsn was never applied to the actual commit watermark. (2) Even after fixing (1) with a durable_lsn-capped store, a newly-promoted leader whose own fsync had not yet caught up to its own WAL tip (group-commit batches fsync on a timer) stayed capped below safe_lsn indefinitely -- nothing re-drove the promotion advance once the flusher eventually synced, since fsync_worker on_durable_advance only re-evaluates the commit engines PENDING per-entry continuations, and a standby has none for replicated entries.",
|
||||
"fix": "New function persist_promotion_advance_committed(p, safe_lsn) (persist.c, declared in persist.h) called from raft_cb_become_leader right after persist_truncate_wal_above: unconditionally advances p->committed_lsn toward safe_lsn (monotonic), forcing a synchronous flusher_flush_once(p) first if durable_lsn < safe_lsn (safe/bounded -- by Raft Leader Completeness the new leaders own WAL tip is always >= safe_lsn, so the bytes are already on disk and only need fdatasync). Reduced but did not fully close the soak failures alone -- see bug-1432 for the second half of the fix.",
|
||||
"tags": [
|
||||
"raft",
|
||||
"promotion",
|
||||
"committed_lsn",
|
||||
"chaotic-soak",
|
||||
"apply-decoupling",
|
||||
"durability"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1432"
|
||||
],
|
||||
"occurrences": 3,
|
||||
"last_seen": "2026-07-10T10:10:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1432",
|
||||
"timestamp": "2026-07-10T10:15:00Z",
|
||||
"error_message": "chaotic HA soak still FAIL (committed_job_loss=8) after bug-1431 force-flush fix; jobs above the conservative 2-voter safe_committed_lsn floor stayed invisible via expect-job for ~3.1s post-promotion, matching the primary heartbeat interval (3000ms) almost exactly",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "ha_start_primary_heartbeat armed the periodic heartbeat via delay_task_ms(interval_ms, -1, primary_heartbeat_fn, c), whose FIRST invocation fires only after interval_ms elapses (standard periodic-timer semantics) -- despite the functions own long-standing comment claiming to start the heartbeat immediately. A freshly-promoted leader therefore waited a full heartbeat interval before ever sending a NOP to a reconnecting/catching-up standby, so it could not learn that standbys true applied_lsn (via the heartbeat-reply stateReport, ha_process_standby_state) for up to 3s. That knowledge gates when catch-up WAL streaming starts and, in turn, the per-entry majority-ack that advances committed_lsn past the elections conservative safe_committed_lsn floor (jobs above that floor were legitimately majority-committed via the OLD primary before it died, per Leader Completeness, but the new leader cannot expose them without fresh quorum corroboration -- a correct Raft Figure-8 safety requirement, not a bug in itself).",
|
||||
"fix": "ha_start_primary_heartbeat now calls primary_heartbeat_fn(c) synchronously once, immediately, before arming the periodic delay_task_ms timer -- matching the functions own documented intent. primary_heartbeat_fn only performs non-blocking async sends (xce_ha_heartbeat_req), so this is safe inline with no thread-role change. Single fix point covers all 3 promotion call sites (raft_cb_become_leader, ha_promote_to_primary, and the third raft-driven-decision-loop site) since all route through ha_start_primary_heartbeat. Closed the gap: 10-round chaotic soak went from committed_job_loss=8 to committed_job_loss=0, committed_divergence=0, result=PASS; 10-round sequential soak + leadership-coverage also PASS (36 jobs, zero loss).",
|
||||
"tags": [
|
||||
"raft",
|
||||
"heartbeat",
|
||||
"promotion",
|
||||
"delaymgr",
|
||||
"chaotic-soak",
|
||||
"latency"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1431"
|
||||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-07-10T10:26:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1433",
|
||||
"timestamp": "2026-07-10T10:59:00Z",
|
||||
"error_message": "controller-invariant-audit (2026-07-10, /xce-audit-invariants) async-I/O-thread-role dimension: bug-1431 round-2 force-flush fix (persist_promotion_advance_committed calling flusher_flush_once, a blocking fdatasync) is reachable via the real multi-node winning-vote-reply dispatch path on a VTASK_LIVENESS-classified vtask worker -- a task class the design documents as never admission-gated / never expected to block. This is the same bug class as bug-1198 (HA_REPL_SEND_SNAPSHOT persist_checkpoint on VTASK_LIVENESS), which was previously closed by offloading the blocking work off that worker class entirely.",
|
||||
"file": "src/controller/persist.c, src/controller/ha.c",
|
||||
"root_cause": "raft_cb_become_leader (the .on_become_leader raft-library callback) is invoked synchronously from within peer_ctrl_msg_handler -> peer_ctrl_dispatch_to_pool -> peer_ctrl_classify (ha.c ~5935-5940), which HARDCODES VTASK_LIVENESS for every inbound peer-control packet including vote replies -- the doc comment above it explicitly states \"everything this handler dispatches is liveness work today\" and was written BEFORE this session added a blocking fdatasync call reachable from that same path. ha_dispatch_election_start/raft_election_start_vtask_entry (the VTASK_LIVE_BLOCK dispatch used for the RDHUP-failover and single-node self-grant election-START triggers) does NOT cover the separate, later \"win the election / become leader\" event, which fires on vote-REPLY arrival via a completely different dispatch chain.",
|
||||
"fix": "Removed the flusher_flush_once force-flush from persist_promotion_advance_committed entirely -- round-3 (durable-capped only, no blocking calls) plus the already-shipped bug-1432 immediate-first-heartbeat fix together close the original visibility-latency gap non-blockingly (confirmed by two live soaks: 6-round chaotic committed_job_loss=0/committed_divergence=0 PASS, 6-round sequential+leadership-coverage PASS). TRACKED RESIDUAL (not fixed this session): persist_apply_pump_run_sync (added earlier this session for the bug-1043-family apply-lag fix, also called from raft_cb_become_leader on the same VTASK_LIVENESS path) does bounded pread() of recently-written WAL segments -- lower practical risk than fdatasync (almost always page-cache-hot) but still, strictly, blocking I/O on a never-blocks task class. Removing it is NOT safe (reopens bug-1043-family job loss). Proper fix needs either reclassifying the winning-vote-reply case in peer_ctrl_classify or dispatching promotion tail onto a fresh VTASK_LIVE_BLOCK vtask -- both larger changes deferred to a dedicated follow-up with its own soak-validation pass. Comments at both call sites corrected to stop falsely claiming VTASK_LIVE_BLOCK safety.",
|
||||
"tags": [
|
||||
"raft",
|
||||
"promotion",
|
||||
"vtask",
|
||||
"thread-role",
|
||||
"liveness",
|
||||
"audit-finding",
|
||||
"bug-1198-class"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1431",
|
||||
"bug-1432",
|
||||
"bug-1198"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-10T10:59:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1434",
|
||||
"timestamp": "2026-07-10T10:56:00Z",
|
||||
"error_message": "controller-invariant-audit (2026-07-10, /xce-audit-invariants) Raft TLA+ checklist dimension: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply can race on persist_t.committed_lsn during the promotion window before role flips to CTRL_PRIMARY, via an unlocked load-then-conditional-store pattern in both functions -- a losing racer could clobber a larger value with a smaller one, regressing committed_lsn (violates the documented commitIndex-monotonic invariant).",
|
||||
"file": "src/controller/persist.c, src/controller/ha.c",
|
||||
"root_cause": "raft_cb_become_leader calls persist_promotion_advance_committed BEFORE setting ha->role = CTRL_PRIMARY (the role store sits ~170 lines later in the function). During that window this node is still nominally non-primary, so ha_apply_wal_catchup_entry (still processing an in-flight WAL catch-up entry on another thread, e.g. if this node was mid-catch-up when it won the election) can legitimately call ha_standby_advance_committed_and_apply concurrently with the promotion path own committed_lsn advance. Both did atomic_load(committed_lsn); if(target<=cur) return; atomic_store(committed_lsn, target) with no lock and no CAS, a classic lost-update race under concurrent writers.",
|
||||
"fix": "Added persist_committed_lsn_advance_atomic(persist_t*, uint64_t target) (persist.c, declared in persist.h) -- a CAS retry loop (atomic_compare_exchange_weak_explicit) that is safe against any number of concurrent callers with no lock required, and can never regress committed_lsn regardless of interleaving. Both persist_promotion_advance_committed and ha_standby_advance_committed_and_apply now route through this shared helper. Validated: build clean (19/19 unit test suites pass), 6-round chaotic soak PASS (0 loss, 0 divergence), 6-round sequential+leadership-coverage soak PASS.",
|
||||
"tags": [
|
||||
"raft",
|
||||
"committed_lsn",
|
||||
"race-condition",
|
||||
"monotonicity",
|
||||
"audit-finding",
|
||||
"CAS"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1431"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-10T10:56:00Z"
|
||||
}
|
||||
]
|
||||
]
|
||||
|
|
@ -99,6 +99,12 @@
|
|||
- (2026-06-14, reserve-lsn-0 TEST FALLOUT) Base-shift broke tests that did a "priming append so the entry lands at LSN >= 1" — that priming was ONLY to escape the old lsn-0 special-case and is now redundant. Fix = REMOVE the priming append so the real entry lands at lsn 1 (test_commit_engine tests 32/34, test_commit_barrier_rollout fact-hook, test_commit_barrier_reply cbrp_setup, test_p2_integration single_node) OR bump a hard-coded `current_lsn==1` assertion to `==2` if the priming must stay (test_egress_filter ef_setup). The ENGINE-LESS walAck path only "worked" via `repl_wait_majority_ack`'s `entry_lsn==0 → return 0` hatch (repl.c:437); with the first entry now lsn 1 it routes to `commit_submit(ce=NULL)` → -EINVAL, so engine-less commit tests MUST wire a commit engine (test_p2_integration test 9: added commit_engine_init(n_peers=3) + repl_set_commit_engine + commit_engine_fini). Tests asserting RELATIVE lsn (lsn_before+1, current_lsn>0, h1.lsn>h0.lsn) or byte-offset truncation math are base-shift-immune.
|
||||
- (2026-07-03, raft-apply-decoupling task 3.3) `test_commit_barrier_reply.c`'s `cbrp_setup` fixture wires job submission's commit continuations through a SEPARATE standalone `commit_engine_t g_hce` (via `commit_engine_init(&g_hce, &g_hc.persist, ...)` + `repl_set_commit_engine`), NOT through `cluster_t`'s own embedded `c->commit` field. This matters because `raft_cb_become_follower`'s demotion branch (ha.c:1395) calls `commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY)` — i.e. it operates on `c->commit`, the field `g_hce` deliberately bypasses. So `raft_cb_become_follower_with_cluster_for_test(&g_hc, term)` CANNOT be used to force a job-submission continuation FAILED in this file's harness — it would touch the wrong (never-initialized, all-zero) engine. `persist_truncate_wal_above(&g_hc.persist, cutoff)` has no such problem: `persist_set_truncate_cb` is wired directly by `commit_engine_init` onto whichever engine you pass it (here, `g_hce`), so truncation is the ONLY viable way to force a deterministic, explicit non-timeout FAILED outcome (`-ENOENT`) for a job-submit continuation in this specific fixture. Added 4 tests exercising this (single + repeated truncation-forced FAILED, repeated timeout-forced FAILED, array mixed-outcome) — all GREEN on first run (no bug found; task 3.2's failure-path guarantee holds). Generic demotion-resolve-continuations-FAILED behavior already has its own dedicated coverage in `test_demotion_convergence.c` (task 1.4/1.5), which uses `ha_promote_to_primary` + the cluster-aware `_for_test` seam against a REAL `c->commit`-shaped fixture — reuse THAT file, not this one, for anything demotion-specific.
|
||||
- (2026-06-13, kiro-impl process) A RED-only test pin is only valid if its OWN impl task can turn it green via the SAME seam the test drives. Misfires caught in review: boot-replay pin RED on an lsn-0 `replay_wal` skip artifact; hooks pin via the `c==NULL` raft shim where `ha_register_persist_hooks(&c->hooksys)` can never run (re-point to the real-cluster `ha_promote_to_primary` seam). Deleting machinery → DELETE the machinery-subject tests (design "no shims, rewritten or deleted not stubbed"), never add file-local no-op stubs. Unity runner prints the PREVIOUS passing test's file:line as a stale prefix on FAIL lines → attribute failures by test NAME.
|
||||
- (2026-07-10, persist_apply_pump_run_sync build+test validation) A bare `grep -i TIMEOUT testlog.txt` is NOT a reliable single-shot hang detector in this suite: it returns 100+ matches every clean run because "timeout" appears in test names (`test_wait_ack_timeout`, `test_heartbeat_timeout_fires_failover`, ...) and in expected WARN-level log lines from tests that deliberately exercise timeout paths (`commit: WAIT deadline expired without commit ... timeout_ms=250`), every one of which is immediately followed by `PASS`. Same false-positive shape as the already-documented `grep -c FAIL` overcount from expected "FAILED" fault-injection log lines (see the `:FAIL:`-anchor entry above) — a plain substring match on an English word that's also load-bearing test vocabulary in an HA/timeout-heavy codebase will always overcount. The AUTHORITATIVE signal for a real meson-level hang is the per-test `result:` field in testlog.txt (`result: exit status 0` = clean; a real hang shows `result: TIMEOUT` or "killed by signal") plus a case-SENSITIVE word-boundary check `grep -noE '\bTIMEOUT\b'` (zero hits = no real timeout verdict anywhere). This is a DIFFERENT axis from the pre-existing 2026-06-13 learning two entries up ("meson wrapper's OWN 300s `Timeout: 1` summary field is not fatal, use the direct binary") — that one is about meson's outer test-runner timing out the whole suite; this one is about naively grepping the word "timeout" inside a clean log. Both must be checked via their authoritative field, never via a bare case-insensitive substring grep.
|
||||
- (2026-07-10, bug-1431/1432) `delay_task_ms(mgr, interval_ms, -1, fn, arg)` fires its FIRST invocation only after `interval_ms` elapses — it is a periodic timer, not "run now then repeat." A comment claiming a delaymgr-scheduled task starts "immediately" is not proof it does; verify against this semantics before trusting it. `ha_start_primary_heartbeat` had exactly this stale claim, costing a full heartbeat interval (3000ms default) of promotion latency before a reconnecting standby's true log position was learned via the heartbeat-reply stateReport. Fix pattern: call the task function synchronously once before arming the periodic `delay_task_ms`, but ONLY if that function does purely non-blocking work (verified here — `primary_heartbeat_fn` only issues async `xce_ha_heartbeat_req` sends).
|
||||
- (2026-07-10, bug-1431) `persist_truncate_wal_above`/`truncate_wal_above_locked` only sets `p->committed_lsn` on branches that perform a REAL truncation. Its common no-op early-return (nothing above cutoff_lsn — the case when a newly-promoted leader's own log already covers the just-computed `safe_committed_lsn`) leaves `committed_lsn` untouched. Any caller that derives a new commit floor and expects `persist_truncate_wal_above` to have applied it (a comment in `raft_cb_become_leader` claimed exactly this) must NOT assume that — use a dedicated advance function instead (`persist_promotion_advance_committed`, durable-capped, monotonic via a CAS loop). **UPDATED 2026-07-10 (bug-1433, controller-invariant-audit):** an earlier version of this fix also force-flushed (`flusher_flush_once`, a blocking `fdatasync`) when durable_lsn lagged — REMOVED, because the real multi-node winning-vote-reply path reaches this function on a `VTASK_LIVENESS`-classified vtask (never expected to block — same bug class as bug-1198). The visibility gap is closed non-blockingly by bug-1432's immediate-heartbeat fix alone; no force-flush needed.
|
||||
- (2026-07-10) A chaotic-HA-soak "COMMITTED JOB LOSS" verdict with `committed_divergence=0` and all jobs present at final settlement is a VISIBILITY-LATENCY gap, not a durability violation: the job was legitimately committed by the old primary before it died, but the new leader's post-election `safe_committed_lsn` (computed only from THIS election's granted-voter sample, which can be conservative when a second voter is mid-restart) is lower until fresh quorum reconfirmation — correct Raft Figure-8 safety, not a bug by itself. Per this project's test philosophy (skill xce-test-ha: "a committed job an elected primary cannot list IS a real loss regardless of eventual reappearance") this is still a hard soak FAIL and must be fixed — but ONLY by shrinking the reconfirmation window (e.g. sending the first post-promotion heartbeat immediately), never by unsafely exposing unconfirmed old-term entries early just to make the check pass.
|
||||
- (2026-07-10, bug-1433, controller-invariant-audit) `peer_ctrl_classify` (ha.c ~5935) hardcodes `VTASK_LIVENESS` for EVERY inbound peer-control packet (vote replies, PreVote replies, WAL/config ACKs) — a task class documented as "never admission-gated," i.e. never expected to block. `raft_cb_become_leader` (the raft library's `.on_become_leader` callback) is invoked synchronously from THIS dispatch path whenever a real multi-node winning vote reply arrives — NOT from the separate `VTASK_LIVE_BLOCK`-dispatched election-*start* path (`ha_dispatch_election_start`, which only covers RDHUP-failover/single-node self-grant). Any blocking call added inside `raft_cb_become_leader` (or anything it calls) is reachable on a worker class the design guarantees never blocks — same bug class as bug-1198 (closed by moving `persist_checkpoint`'s snapshot-send fdatasync off VTASK_LIVENESS entirely). Before adding ANY blocking call to the promotion path, trace peer_ctrl_classify's actual dispatch, don't trust an existing nearby comment claiming "already on VTASK_LIVE_BLOCK" — one was WRONG (see bug-1433 in buglog.json for the full trace). `persist_apply_pump_run_sync` still has this residual (bounded `pread`, lower risk than `fdatasync`, not fixed this session — see buglog.json bug-1433 for why it can't simply be removed).
|
||||
- (2026-07-10, bug-1434, controller-invariant-audit) `persist_promotion_advance_committed` and `ha_standby_advance_committed_and_apply` both did an unlocked `load; if(target<=cur) return; store` on `p->committed_lsn` — safe ONLY if truly single-writer, but `raft_cb_become_leader` calls the former BEFORE setting `role = CTRL_PRIMARY` (~170 lines later), so a still-in-flight WAL catch-up entry on another thread can call the latter concurrently during that window — classic lost-update race that can regress `committed_lsn` (monotonicity violation). General lesson: when two independent call sites do "monotonic-max store to a shared atomic, no lock" and you can't prove they're strictly serialized by role/state gating (gating checked AT CALL TIME can still race with a WINDOW where the gate value itself is about to change), use a CAS retry loop (`persist_committed_lsn_advance_atomic`), not a bare load-then-store — it's safe under ANY interleaving with zero lock-ordering risk.
|
||||
|
||||
## Do-Not-Repeat
|
||||
|
||||
|
|
|
|||
|
|
@ -391,20 +391,27 @@ int config_activate_committed(cluster_t *c)
|
|||
* non-promotable) for that upgrade until reboot — a §7.9 liveness violation.
|
||||
*
|
||||
* The fix: when commit advances WHILE this member is serve-fenced, schedule a
|
||||
* one-shot OFF-IOMGR delaymgr task that re-runs config_activate_committed. The
|
||||
* delaymgr worker runs on a thread allowed to block on the disk reads
|
||||
* (scan_latest_committed_marker pread + load_primary_config). Once the NEW marker
|
||||
* is in the committed prefix AND the locally-held bytes hash-match it, the retry
|
||||
* activates and CLEARS the fence; a genuine still-MISMATCH (or no-bytes-yet) keeps
|
||||
* the fence and the next commit-advance reschedules. -ENOENT still falls back
|
||||
* without fencing (the no-marker path is benign). Idempotent + self-limiting: the
|
||||
* scheduler is a no-op once unfenced, so there is no busy-spin. */
|
||||
* one-shot task that re-runs config_activate_committed off the iomgr ACK thread.
|
||||
* The delaymgr thread itself must never block (ha-async-io.md thread-role
|
||||
* contract — it also drives heartbeats and every election timeout), so the
|
||||
* one-shot delaymgr task is only a trigger: it hands the actual blocking disk
|
||||
* reads (scan_latest_committed_marker pread + load_primary_config) to a
|
||||
* VTASK_LIVE_BLOCK vtask and returns immediately (2026-07-09 correction — this
|
||||
* comment previously and incorrectly claimed the delaymgr worker itself was
|
||||
* "allowed to block on the disk reads"; that predates the corrected contract
|
||||
* ha-async-io.md documents). Once the NEW marker is in the committed prefix AND
|
||||
* the locally-held bytes hash-match it, the retry activates and CLEARS the
|
||||
* fence; a genuine still-MISMATCH (or no-bytes-yet) keeps the fence and the next
|
||||
* commit-advance reschedules. -ENOENT still falls back without fencing (the
|
||||
* no-marker path is benign). Idempotent + self-limiting: the scheduler is a
|
||||
* no-op once unfenced, so there is no busy-spin. */
|
||||
|
||||
/* config_activate_retry_fn: the delaymgr task body (runs OFF the iomgr ACK
|
||||
* thread). Re-runs config_activate_committed so a freshly-committed config marker
|
||||
* is activated — and a stale serve-fence cleared — once its out-of-band bytes
|
||||
* verify. `arg` is the cluster_t* (passed directly; no heap ctx to free, so the
|
||||
* delaymgr node-killer that frees only the delay_task_t never leaks). */
|
||||
/* config_activate_retry_fn: re-runs config_activate_committed so a freshly-
|
||||
* committed config marker is activated — and a stale serve-fence cleared —
|
||||
* once its out-of-band bytes verify. Does the actual blocking disk I/O; callers
|
||||
* must invoke it off the delaymgr thread (see config_activate_retry_vtask_entry
|
||||
* / config_schedule_activate_retry below). Public so tests can drive it
|
||||
* directly without a vtask pool. */
|
||||
int config_activate_retry_fn(void *arg)
|
||||
{
|
||||
cluster_t *c = (cluster_t *)arg;
|
||||
|
|
@ -425,8 +432,56 @@ int config_activate_retry_fn(void *arg)
|
|||
return rc;
|
||||
}
|
||||
|
||||
/* config_schedule_activate_retry: enqueue config_activate_retry_fn on the
|
||||
* delaymgr (NON-BLOCKING — delay_task_ms only does ordq_put). Called from the
|
||||
/* config_activate_retry_vtask_entry: vtask entry wrapping config_activate_
|
||||
* retry_fn so its blocking disk I/O runs on a vtask worker, never inline on
|
||||
* the single delaymgr thread. */
|
||||
static int config_activate_retry_vtask_entry(vtask_t *task)
|
||||
{
|
||||
cluster_t *c = (cluster_t *)task->data->u.generic;
|
||||
config_activate_retry_fn(c);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* config_activate_retry_delaymgr_trampoline: the delaymgr task body. Delaymgr
|
||||
* must only trigger + enqueue, never block — dispatch the actual retry work to
|
||||
* a vtask (VTASK_LIVE_BLOCK, the same class the vote-grant term fsync uses,
|
||||
* R4.6) and return immediately. Fails loud and runs inline (stay alive, never
|
||||
* silently drop a scheduled retry) if the pool is unavailable. */
|
||||
static int config_activate_retry_delaymgr_trampoline(void *arg)
|
||||
{
|
||||
cluster_t *c = (cluster_t *)arg;
|
||||
if (!c)
|
||||
return -EINVAL;
|
||||
|
||||
vtaskmgr_t *pool = &c->vtaskmgr;
|
||||
vtask_data_t *data = calloc(1, sizeof(*data));
|
||||
vtask_t *task = vtask_alloc();
|
||||
|
||||
if (data == NULL || task == NULL) {
|
||||
WARN("config commit-advance retry: vtask alloc failed — running "
|
||||
"inline (stay alive)");
|
||||
free(data);
|
||||
free(task);
|
||||
return config_activate_retry_fn(c);
|
||||
}
|
||||
|
||||
data->vtype = VTASK_GENERIC;
|
||||
data->u.generic = c;
|
||||
vtask_init(task, (vtask_func_t)config_activate_retry_vtask_entry, data);
|
||||
|
||||
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
|
||||
WARN("config commit-advance retry: vtask enqueue failed — running "
|
||||
"inline (stay alive)");
|
||||
free(data);
|
||||
free(task);
|
||||
return config_activate_retry_fn(c);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* config_schedule_activate_retry: enqueue the retry trampoline on the delaymgr
|
||||
* (NON-BLOCKING — delay_task_ms only does ordq_put; the trampoline itself
|
||||
* offloads the blocking work to a vtask, see above). Called from the
|
||||
* commit-advance path (iomgr ACK thread) so activation's disk I/O runs OFF that
|
||||
* thread (ha-async-io invariant). No-op when the member is NOT serve-fenced (so
|
||||
* the steady state schedules nothing) or when the delaymgr is not running (the
|
||||
|
|
@ -443,7 +498,8 @@ int config_schedule_activate_retry(cluster_t *c)
|
|||
|
||||
/* One-shot, minimal delay: run promptly but off the iomgr ACK thread. */
|
||||
int rc = delay_task_ms(&c->delaymgr, 1, 1,
|
||||
(delay_task_entry_fn_t)config_activate_retry_fn, c);
|
||||
(delay_task_entry_fn_t)config_activate_retry_delaymgr_trampoline,
|
||||
c);
|
||||
if (rc < 0)
|
||||
WARN("config commit-advance retry: delay_task_ms failed: %d — fence will "
|
||||
"clear on a later commit advance or config-push", rc);
|
||||
|
|
@ -2540,12 +2596,21 @@ typedef struct {
|
|||
} ha_catchup_continuation_ctx_t;
|
||||
|
||||
/*
|
||||
* ha_catchup_continuation_fn — delaymgr task body (runs OFF the iomgr walAck
|
||||
* branch, D6/D1). Re-validate the endpoint against the registry, re-derive
|
||||
* target, stream [acked_lsn+1, target], and clear catchup_inflight on EVERY exit
|
||||
* path (success, ep-gone, nothing-to-stream) so a stuck flag never blocks future
|
||||
* continuations (Req 5.2). Frees its own ctx (body-frees-own-ctx pattern,
|
||||
* mirroring ha_demotion_convergence_fn).
|
||||
* ha_catchup_continuation_fn — the real catch-up work. Re-validate the
|
||||
* endpoint against the registry, re-derive target, stream [acked_lsn+1,
|
||||
* target], and clear catchup_inflight on EVERY exit path (success, ep-gone,
|
||||
* nothing-to-stream) so a stuck flag never blocks future continuations (Req
|
||||
* 5.2). Frees its own ctx (body-frees-own-ctx pattern, mirroring
|
||||
* ha_demotion_convergence_run).
|
||||
*
|
||||
* Runs as a delaymgr task body (see ha_schedule_catchup_continuation below) —
|
||||
* must run off the iomgr walAck thread, since catchup_stream_range ->
|
||||
* ha_stream_wal_to_standby does blocking WAL file reads + socket writes.
|
||||
* (2026-07-10: a VTASK_LIVE_BLOCK dispatch was attempted here to also get it
|
||||
* off the delaymgr thread per ha-async-io.md's thread-role contract, but it
|
||||
* deadlocked the controller_unit_test harness — reverted pending a proper
|
||||
* fix; tracked as a known, still-open thread-role violation alongside the
|
||||
* other raft-drive-async P2 items.)
|
||||
*/
|
||||
int ha_catchup_continuation_fn(void *arg)
|
||||
{
|
||||
|
|
@ -2587,10 +2652,11 @@ clear_and_done:
|
|||
}
|
||||
|
||||
/*
|
||||
* ha_schedule_catchup_continuation — enqueue one one-shot delaymgr task running
|
||||
* ha_catchup_continuation_fn. The single-flight flag is set by the caller
|
||||
* (ha_walack_on_ack) before this call; on a scheduling failure this clears it so
|
||||
* the slot does not stay stuck. Returns 0 on success, negative on failure.
|
||||
* ha_schedule_catchup_continuation — enqueue one one-shot delaymgr task
|
||||
* running ha_catchup_continuation_fn directly. The single-flight flag is set
|
||||
* by the caller (ha_walack_on_ack) before this call; on a scheduling failure
|
||||
* this clears it so the slot does not stay stuck. Returns 0 on success,
|
||||
* negative on failure.
|
||||
*/
|
||||
static int ha_schedule_catchup_continuation(cluster_t *c, repl_ep_t ep,
|
||||
int peer_idx, uint64_t acked_lsn)
|
||||
|
|
@ -2610,8 +2676,9 @@ static int ha_schedule_catchup_continuation(cluster_t *c, repl_ep_t ep,
|
|||
|
||||
/* One-shot (runcount=1), minimal delay: streaming reads WAL files and must
|
||||
* run off the iomgr walAck thread. delay_task_ms only enqueues (ordq_put);
|
||||
* the delaymgr worker runs it. In XCE_TESTING the fixture drives the queued
|
||||
* task synchronously via its stored entry/args. */
|
||||
* the delaymgr worker runs ha_catchup_continuation_fn itself. In
|
||||
* XCE_TESTING the fixture drives the queued task synchronously via its
|
||||
* stored entry/args. */
|
||||
int rc = delay_task_ms(&c->delaymgr, 1, 1,
|
||||
(delay_task_entry_fn_t)ha_catchup_continuation_fn,
|
||||
ctx);
|
||||
|
|
|
|||
|
|
@ -108,9 +108,26 @@ extern int config_schedule_activate_retry(cluster_t *c);
|
|||
static void on_vote_connected(endpoint_t *ep);
|
||||
static void on_pre_vote_connected(endpoint_t *ep);
|
||||
static int vote_ep_err_handler(void *ep_raw);
|
||||
/* bug-1423: forward-declared so raft_cb_send_vote_request's connect/monitor
|
||||
* failure branches can route a synthetic "no vote" reply through the SAME
|
||||
* production path a real reply uses (see vote_reply_dispatch's own comment,
|
||||
* below, for why this must never be the legacy handle_request_vote_reply). */
|
||||
static void vote_reply_dispatch(ha_state_t *ha, ha_elect_ctx_t *ctx, int idx,
|
||||
uint64_t term, int vote_granted,
|
||||
uint64_t peer_tag, uint64_t peer_lsn);
|
||||
/* bug-1423 (PreVote twin): same rationale as vote_reply_dispatch above, for
|
||||
* raft_cb_send_pre_vote_request's connect/monitor failure branches. */
|
||||
static void pre_vote_reply_dispatch(ha_state_t *ha, cluster_t *c,
|
||||
int peer_idx, int granted);
|
||||
/* bug-1197: defer a vote-ep teardown onto the per-ep FIFO of the vtask pool so
|
||||
* it runs strictly after the last live use of the ep (flag-ON loser path). */
|
||||
static void vote_ep_teardown_async(ha_state_t *ha, endpoint_t *ep);
|
||||
/* 2026-07-10 controller-invariant audit: forward-declared so the catch-up
|
||||
* ingest path (ha_apply_wal_catchup_entry, earlier in this file) can kick the
|
||||
* standby apply pump against the already-known commit position without
|
||||
* waiting for the next heartbeat. Defined near ha_observe_heartbeat_committed. */
|
||||
static void ha_standby_advance_committed_and_apply(cluster_t *c,
|
||||
uint64_t learned_committed_lsn);
|
||||
/* ha-single-leader-role task 2.2: shared promotion seam — every road to
|
||||
* PRIMARY registers the persist hooks and arms the own-term gate. */
|
||||
static void ha_register_persist_hooks(cluster_t *c);
|
||||
|
|
@ -222,15 +239,36 @@ static int raft_cb_send_vote_request(void *ctx, int peer_idx, uint64_t term)
|
|||
DEFAULT_CONNECT_TIMEOUT_MS);
|
||||
if (rc < 0) {
|
||||
endpoint_unref(ep); /* bug-1298: refcnt 1->0 → fini+free (not monitored) */
|
||||
/* bug-1423: a connect failure (e.g. peer dead/unreachable) must still
|
||||
* register as a "no vote" reply through the SAME path a real reply
|
||||
* uses. Without this, raft_state_t.replies_received never reaches
|
||||
* n_peers-1 whenever any peer is unreachable, so the library's
|
||||
* "everyone's replied, decide now" fast path (raft.c "Step 6:
|
||||
* all-replied-no-quorum") is structurally unreachable — recovery then
|
||||
* depends solely on the raw election timeout, observed to leave a
|
||||
* candidate stuck in CTRL_ELECTING far longer than expected in a
|
||||
* 3-node cluster with one node down. Test mode (c==NULL) keeps the
|
||||
* legacy handle_request_vote_reply call that existing unit tests
|
||||
* depend on; production routes through vote_reply_dispatch, the same
|
||||
* call a genuine reply makes (ha.c vote_reply_dispatch). */
|
||||
if (!c)
|
||||
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
|
||||
else
|
||||
vote_reply_dispatch(ha, ectx, votes_idx, term, /*vote_granted=*/0, 0, 0);
|
||||
return 1;
|
||||
}
|
||||
|
||||
rc = iomgr_monitor(ep);
|
||||
if (rc < 0) {
|
||||
endpoint_unref(ep); /* bug-1298: fini frees ep->data (vctx), then frees ep */
|
||||
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
|
||||
/* bug-1423: same fix as the connect-failure branch above — this was
|
||||
* unconditionally calling the legacy handle_request_vote_reply even in
|
||||
* production, bypassing raft_on_vote_reply (the sole production
|
||||
* promotion driver per vote_reply_dispatch's own comment). */
|
||||
if (!c)
|
||||
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
|
||||
else
|
||||
vote_reply_dispatch(ha, ectx, votes_idx, term, /*vote_granted=*/0, 0, 0);
|
||||
return 1;
|
||||
}
|
||||
|
||||
|
|
@ -332,10 +370,30 @@ static int raft_cb_send_pre_vote_request(void *ctx, int peer_idx,
|
|||
|
||||
int rc = endpoint_connect(ep, peer->hostname, peer->port,
|
||||
DEFAULT_CONNECT_TIMEOUT_MS);
|
||||
if (rc < 0) { endpoint_unref(ep); return 1; } /* bug-1298: fini+free */
|
||||
if (rc < 0) {
|
||||
endpoint_unref(ep); /* bug-1298: fini+free */
|
||||
/* bug-1423 (PreVote twin): a connect failure (dead/unreachable peer)
|
||||
* must still register as a "not granted" reply with the raft library's
|
||||
* PreVote tally, same as the real-vote send path. PreVote's own round
|
||||
* timeout (ha_prevote_round_timeout_fn) is an unconditional fallback
|
||||
* that always resolves the round regardless, so this gap is not a
|
||||
* permanent-deadlock class bug like the real-vote one was — but
|
||||
* leaving it means every round against a down peer wastes the full
|
||||
* round timeout instead of concluding as soon as the live peer
|
||||
* replies. */
|
||||
if (c)
|
||||
pre_vote_reply_dispatch(ha, c, peer_idx, /*granted=*/0);
|
||||
return 1;
|
||||
}
|
||||
|
||||
rc = iomgr_monitor(ep);
|
||||
if (rc < 0) { endpoint_unref(ep); return 1; } /* bug-1298: fini+free */
|
||||
if (rc < 0) {
|
||||
endpoint_unref(ep); /* bug-1298: fini+free */
|
||||
/* bug-1423 (PreVote twin): same fix as the connect-failure branch above. */
|
||||
if (c)
|
||||
pre_vote_reply_dispatch(ha, c, peer_idx, /*granted=*/0);
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -482,15 +540,64 @@ static void raft_cb_cancel_timeout(void *ctx)
|
|||
ha->raft_election_timeout_ctx = NULL;
|
||||
}
|
||||
|
||||
/* raft_election_start_vtask_entry: vtask entry wrapping raft_check_and_start_
|
||||
* election. raft_start_election's on_persist callback (raft_cb_persist ->
|
||||
* ha_persist_term) does a synchronous fsync -- must never run inline on the
|
||||
* delaymgr thread (ha-async-io.md thread-role contract: delaymgr only fires
|
||||
* timers and triggers work, then returns; it never blocks, since the same
|
||||
* thread also drives heartbeats and every other election timeout). */
|
||||
static int raft_election_start_vtask_entry(vtask_t *task)
|
||||
{
|
||||
ha_state_t *ha = (ha_state_t *)task->data->u.generic;
|
||||
raft_check_and_start_election(&ha->raft);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ha_dispatch_election_start: offload raft_check_and_start_election onto a
|
||||
* vtask (VTASK_LIVE_BLOCK -- the same class already used for the vote-GRANT
|
||||
* term fsync, R4.6) so its persist-before-campaign fsync never blocks the
|
||||
* delaymgr thread. On pool unavailability/enqueue failure, fail loud and run
|
||||
* inline anyway (stay alive, never silently drop a scheduled election --
|
||||
* mirrors persist_apply_pump_kick's and vote_ep_teardown_enqueue's fallback). */
|
||||
static void ha_dispatch_election_start(ha_state_t *ha)
|
||||
{
|
||||
vtaskmgr_t *pool = (vtaskmgr_t *)ha->pool;
|
||||
vtask_data_t *data = calloc(1, sizeof(*data));
|
||||
vtask_t *task = vtask_alloc();
|
||||
|
||||
if (pool == NULL || data == NULL || task == NULL) {
|
||||
WARN("ha_dispatch_election_start: vtask unavailable (pool=%p data=%p "
|
||||
"task=%p) — starting election inline (stay alive)",
|
||||
(void *)pool, (void *)data, (void *)task);
|
||||
free(data);
|
||||
free(task);
|
||||
raft_check_and_start_election(&ha->raft);
|
||||
return;
|
||||
}
|
||||
|
||||
data->vtype = VTASK_GENERIC;
|
||||
data->u.generic = ha;
|
||||
vtask_init(task, (vtask_func_t)raft_election_start_vtask_entry, data);
|
||||
|
||||
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
|
||||
WARN("ha_dispatch_election_start: vtask enqueue failed — starting "
|
||||
"election inline (stay alive)");
|
||||
free(data);
|
||||
free(task);
|
||||
raft_check_and_start_election(&ha->raft);
|
||||
}
|
||||
}
|
||||
|
||||
/* raft_rdhup_delay_fn: delaymgr callback for RDHUP-triggered delayed election.
|
||||
* Owns its ctx (election_timeout_ctx_t) — frees it on exit.
|
||||
* Calls raft_check_and_start_election to start election only if not already voted. */
|
||||
* Dispatches the actual election start off the delaymgr thread (see
|
||||
* ha_dispatch_election_start) only if this node hasn't already voted. */
|
||||
static int raft_rdhup_delay_fn(void *arg)
|
||||
{
|
||||
election_timeout_ctx_t *t = (election_timeout_ctx_t *)arg;
|
||||
ha_state_t *ha = t->ha;
|
||||
if (!atomic_load(&t->cancelled))
|
||||
raft_check_and_start_election(&ha->raft);
|
||||
ha_dispatch_election_start(ha);
|
||||
free(t);
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -506,9 +613,11 @@ static void raft_cb_start_election_after_delay(void *ctx, uint32_t delay_ms)
|
|||
cluster_t *c = ha->raft_c;
|
||||
if (!c) return; /* test mode */
|
||||
|
||||
/* Priority 0 → delay_ms == 0: highest-priority node starts immediately. */
|
||||
/* Priority 0 → delay_ms == 0: highest-priority node starts immediately.
|
||||
* Dispatched off the calling thread (see ha_dispatch_election_start) since
|
||||
* this callback's caller is not guaranteed to be vtask-safe. */
|
||||
if (delay_ms == 0) {
|
||||
raft_check_and_start_election(&ha->raft);
|
||||
ha_dispatch_election_start(ha);
|
||||
return;
|
||||
}
|
||||
|
||||
|
|
@ -870,10 +979,19 @@ static int promotion_fence_timeout_fn(void *arg)
|
|||
* unconditionally for the same reason — see persist.c). */
|
||||
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
|
||||
|
||||
/* self_restart_pending: same unconditional, never-live-cleared refusal as
|
||||
* apply_divergent — see the field's doc comment in ha.h. This purely
|
||||
* structural re-check (durable floor vs HWM) must not release a fence set
|
||||
* because a demoted ex-primary's execv() failed; that node's WAL state can
|
||||
* look fully caught-up while its in-memory state is still stale. */
|
||||
bool self_restart_pending =
|
||||
atomic_load_explicit(&ha->self_restart_pending, memory_order_acquire);
|
||||
|
||||
/* Caught up: not truncated AND durable position reached the boundary AND
|
||||
* the current apply verdict is not DIVERGENT → release the fence + clear
|
||||
* the sticky flag; the member is eligible on its next election (Req 2.6). */
|
||||
if (!truncated && !below_hwm && !apply_divergent) {
|
||||
* the current apply verdict is not DIVERGENT AND no self-restart is
|
||||
* pending → release the fence + clear the sticky flag; the member is
|
||||
* eligible on its next election (Req 2.6). */
|
||||
if (!truncated && !below_hwm && !apply_divergent && !self_restart_pending) {
|
||||
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
|
||||
persist_clear_recovery_unrecoverable(&c->persist);
|
||||
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
|
||||
|
|
@ -1046,10 +1164,20 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
|
|||
uint64_t trunc_floor =
|
||||
ha_promotion_truncate_floor(safe_lsn, durable_tip);
|
||||
persist_truncate_wal_above(&c->persist, trunc_floor);
|
||||
/* 2026-07-10 (chaotic-soak finding): persist_truncate_wal_above
|
||||
* only advances persist->committed_lsn on the branches that
|
||||
* perform a REAL truncation; its no-op path (this leader's own
|
||||
* log already covers safe_lsn -- the common case) leaves
|
||||
* committed_lsn untouched at its pre-election value, stalling
|
||||
* commit-gated apply until the normal ack-driven commit path
|
||||
* organically catches up seconds later. See
|
||||
* persist_promotion_advance_committed's doc comment. */
|
||||
persist_promotion_advance_committed(&c->persist, safe_lsn);
|
||||
}
|
||||
/* Task 4.2 (ha-repl-hardening): set committed_lsn = safe_lsn.
|
||||
* persist_truncate_wal_above already sets c->persist.committed_lsn
|
||||
* in the production path (c != NULL). Also update
|
||||
* persist_promotion_advance_committed (above) advances
|
||||
* c->persist.committed_lsn in the production path (c != NULL),
|
||||
* durable-capped and monotonic. Also update
|
||||
* ha->last_known_committed_lsn so the test-mode decision loop
|
||||
* (XCE_TESTING path in standby heartbeat reply handler) and any
|
||||
* code reading ha->last_known_committed_lsn see the safe commit
|
||||
|
|
@ -1098,6 +1226,17 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
|
|||
* recomputed by a fresh boot-time persist_recover() replay. */
|
||||
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
|
||||
|
||||
/* self_restart_pending (2026-07-10, ha-convergence-recovery Req 1.4
|
||||
* supersession): a demoted ex-primary whose execv() re-exec failed
|
||||
* must never become primary again while carrying its stale pre-
|
||||
* demotion in-memory state — same unconditional-refuse treatment as
|
||||
* apply_divergent, and for the same reason: this node's WAL/durability
|
||||
* state can look perfectly ELIGIBLE while the in-memory state the
|
||||
* restart exists to discard is still live. No live-clear path exists;
|
||||
* only a successful restart (a fresh process) clears it. */
|
||||
bool self_restart_pending =
|
||||
atomic_load_explicit(&ha->self_restart_pending, memory_order_acquire);
|
||||
|
||||
/* A healthy peer is reachable for catch-up iff some voter granted with a
|
||||
* live endpoint survives this election (these become the standby
|
||||
* channels promoted below). */
|
||||
|
|
@ -1127,30 +1266,35 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
|
|||
truncated, durable_floor, promotion_safe_lsn,
|
||||
peer_reachable, /*catch_up_succeeded=*/false);
|
||||
|
||||
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent) {
|
||||
/* ELIGIBLE, not sticky-unrecoverable, AND not apply-divergent:
|
||||
* release the fence (a caught-up member regains eligibility,
|
||||
* Req 2.6), clear the sticky flag if set, and cancel any pending
|
||||
* re-alert timer. Proceed to authoritative promotion below. */
|
||||
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent
|
||||
&& !self_restart_pending) {
|
||||
/* ELIGIBLE, not sticky-unrecoverable, not apply-divergent, AND no
|
||||
* pending self-restart: release the fence (a caught-up member
|
||||
* regains eligibility, Req 2.6), clear the sticky flag if set, and
|
||||
* cancel any pending re-alert timer. Proceed to authoritative
|
||||
* promotion below. */
|
||||
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
|
||||
persist_clear_recovery_unrecoverable(&c->persist);
|
||||
ha_cancel_promotion_fence_timer(ha);
|
||||
} else {
|
||||
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, or apply-divergent:
|
||||
* refuse promotion — do NOT become CTRL_PRIMARY. Fence the member
|
||||
* and arm the catch-up re-alert timer. The member STAYS ALIVE,
|
||||
* non-promotable, retrying catch-up (Req 2.2/2.4). NEVER
|
||||
* self-exit, never fence silently. An apply-divergent member is
|
||||
* refused unconditionally (Req 8.4) — no peer/HWM condition can
|
||||
* clear that trigger; only a fresh, clean restart replay can. */
|
||||
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, apply-divergent, or
|
||||
* self-restart-pending: refuse promotion — do NOT become
|
||||
* CTRL_PRIMARY. Fence the member and arm the catch-up re-alert
|
||||
* timer. The member STAYS ALIVE, non-promotable, retrying catch-up
|
||||
* (Req 2.2/2.4). NEVER self-exit, never fence silently. An
|
||||
* apply-divergent or self-restart-pending member is refused
|
||||
* unconditionally (Req 8.4; ha-convergence-recovery Req 1.4) — no
|
||||
* peer/HWM condition can clear either trigger; only a fresh boot
|
||||
* (clean restart replay, or a successful re-exec) can. */
|
||||
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d "
|
||||
"apply_divergent=%d) — recovery_truncated=%d durable_floor=%"
|
||||
PRIu64 " cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
|
||||
"apply_divergent=%d self_restart_pending=%d) — "
|
||||
"recovery_truncated=%d durable_floor=%" PRIu64
|
||||
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
|
||||
"arming catch-up re-alert window, staying ALIVE "
|
||||
"(ha-commit-durability Req 2.2/2.4; raft-apply-decoupling "
|
||||
"Req 8.4)",
|
||||
(int)gate, unrecoverable, apply_divergent, truncated,
|
||||
durable_floor, promotion_safe_lsn, peer_reachable);
|
||||
"Req 8.4; ha-convergence-recovery Req 1.4)",
|
||||
(int)gate, unrecoverable, apply_divergent, self_restart_pending,
|
||||
truncated, durable_floor, promotion_safe_lsn, peer_reachable);
|
||||
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
|
||||
promotion_safe_lsn);
|
||||
return;
|
||||
|
|
@ -1201,6 +1345,52 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
|
|||
INFO("HA FSM (raft): → CTRL_PRIMARY (term=%" PRIu64
|
||||
", first_boot=%d)", term, is_first_boot);
|
||||
|
||||
/* 2026-07-10 (chaotic-soak regression finding, bug-1043 family recurrence):
|
||||
* a node promoted from standby can carry an apply-lag backlog -- entries
|
||||
* already committed per c->persist.committed_lsn (just finalized above by
|
||||
* the truncate-clamp / safe_lsn assignment) but never materialized into
|
||||
* the job/queue table, because standby-side materialization is now
|
||||
* commit-gated and only kicked by heartbeat consumption or catch-up
|
||||
* ingest (ha_standby_advance_committed_and_apply) -- BOTH of which stop
|
||||
* firing the instant this node becomes CTRL_PRIMARY (ha_consume_heartbeat_
|
||||
* committed explicitly skips primaries). Without draining here, that
|
||||
* backlog sits unapplied: expect-job/joblist on the new primary would
|
||||
* report a genuinely committed (WAL-consistent, walcompare-clean) job as
|
||||
* missing.
|
||||
*
|
||||
* SYNCHRONOUS on purpose (persist_apply_pump_run_sync, not the async
|
||||
* persist_apply_pump_kick): the async kick only ENQUEUES the drain onto a
|
||||
* vtask and returns immediately, so this function (and thus promotion)
|
||||
* could complete -- and the node could start answering expect-job/joblist
|
||||
* queries as primary -- before the vtask worker actually finished
|
||||
* applying the backlog. A live chaotic HA soak caught exactly that race
|
||||
* (2026-07-10): the async version still lost committed jobs on a freshly
|
||||
* promoted primary.
|
||||
*
|
||||
* THREAD-CONTEXT CORRECTION (2026-07-10, controller-invariant-audit): this
|
||||
* comment previously claimed promotion "already runs on a VTASK_LIVE_BLOCK
|
||||
* vtask" -- that is TRUE only for the single-node self-grant and RDHUP-
|
||||
* failover election-start paths (ha_dispatch_election_start). It is FALSE
|
||||
* for the common real multi-node case: a winning RequestVoteReply is
|
||||
* dispatched via peer_ctrl_classify, which hardcodes VTASK_LIVENESS (never
|
||||
* admission-gated, documented as never expected to block -- see the
|
||||
* bug-1198 hazard box above peer_ctrl_classify's definition, which closed
|
||||
* this EXACT class of bug for HA_REPL_SEND_SNAPSHOT's persist_checkpoint).
|
||||
* persist_apply_pump_run_sync's pread()s of recently-written WAL segments
|
||||
* are lower practical risk than a bare fdatasync (almost always page-cache
|
||||
* hot, since the entries were just appended) but are still, strictly,
|
||||
* blocking I/O on a task class the design says never blocks. This residual
|
||||
* gap is tracked (see buglog.json) rather than fixed inline here: closing
|
||||
* it properly means either reclassifying the winning-vote-reply case in
|
||||
* peer_ctrl_classify or dispatching promotion's tail onto a fresh
|
||||
* VTASK_LIVE_BLOCK vtask, both larger changes needing their own dedicated
|
||||
* soak-validation pass. Do NOT remove this call to "fix" the classification
|
||||
* gap -- it is what closes the bug-1043-family apply-lag/job-loss bug;
|
||||
* removing it reopens that regression. The backlog itself stays inherently
|
||||
* small (only entries committed since the last heartbeat-driven kick). */
|
||||
if (c)
|
||||
persist_apply_pump_run_sync(&c->persist);
|
||||
|
||||
/* Req 2.1 (ha-repl-hardening): re-initialize per-standby convergence
|
||||
* tracking for THIS primary term. Stale converged flags from a
|
||||
* previous term (demote → re-elect cycle) must not satisfy the
|
||||
|
|
@ -2132,9 +2322,11 @@ void ha_fini(ha_state_t *ha)
|
|||
* Two modes (Req 5.3, 5.5):
|
||||
*
|
||||
* Batch mode (entry->n_entries > 0):
|
||||
* Iterate entry->entries[0..n_entries-1] in order. Apply each sub-entry
|
||||
* individually via apply_wal_entry(). NULL sub-entries are treated as
|
||||
* corrupt and skipped (Req 3.3). After ALL sub-entries are processed,
|
||||
* Iterate entry->entries[0..n_entries-1] in order. Persist each
|
||||
* sub-entry individually (entity materialization happens later, once
|
||||
* committed — see ha_standby_advance_committed_and_apply). NULL
|
||||
* sub-entries are treated as corrupt and skipped (Req 3.3). After ALL
|
||||
* sub-entries are processed,
|
||||
* advance applied_lsn to the last-successfully-applied lsn so the next
|
||||
* heartbeat reply reflects post-batch state (Req 5.3).
|
||||
*
|
||||
|
|
@ -2166,11 +2358,10 @@ static void ha_advance_applied_lsn(ha_state_t *ha, uint64_t new_lsn)
|
|||
}
|
||||
}
|
||||
|
||||
/* ha_apply_single_entry: apply one decoded wal_catchup_entry_t sub-entry.
|
||||
* Payload bytes (raw_payload/raw_payload_len, extracted by xce_decode)
|
||||
* are passed through to apply_wal_entry so entity state is restored —
|
||||
* not just the LSN chain (task 7.3).
|
||||
* Returns 0 on success, non-zero on error (caller logs and skips).
|
||||
/* ha_apply_single_entry: durably persist one decoded wal_catchup_entry_t
|
||||
* sub-entry (name retained for history; despite the name it no longer
|
||||
* materializes entity state here — see ha_standby_advance_committed_and_apply,
|
||||
* 2026-07-10). Returns 0 on success, non-zero on error (caller logs and skips).
|
||||
*
|
||||
* raft-ha-core-safety task 6.2 — TERM-BLIND DIRECT-PERSIST BYPASS, SAFE-BY-
|
||||
* GUARD (documented, not structurally retired): this function does NOT go
|
||||
|
|
@ -2229,32 +2420,24 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
|
|||
return prc;
|
||||
}
|
||||
|
||||
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): defer the in-memory
|
||||
* entity apply while this node's OWN boot-time persist_recover() replay+
|
||||
* verification pass is still running. The entry is already durably
|
||||
* persisted above (persist_wal_write_replicated, fsync'd) — only the
|
||||
* ACK and applied-lsn advancement depend on that, per the persist-
|
||||
* before-apply-before-ACK contract this function's own header comment
|
||||
* documents, and bug-1043's precedent of decoupling durability/ACK from
|
||||
* entity materialization. Applying here too, concurrently with
|
||||
* replay_wal()'s single-threaded, unlocked bookkeeping (last_applied_lsn)
|
||||
* and the live job table it reads, can produce a false APPLY_DIVERGENT
|
||||
* verdict on a perfectly healthy boot (confirmed live). No entry is
|
||||
* lost: this node's own in-progress replay_wal() pass will pick it up
|
||||
* directly off disk if it lands within that pass's scan, and
|
||||
* persist_recover()'s own deferred catch-up kick (persist_apply_pump_
|
||||
* kick, issued right before it returns) covers it unconditionally
|
||||
* either way, since apply_pump_read_entry() does a fresh on-disk lookup
|
||||
* every call rather than trusting any earlier directory listing. */
|
||||
if (!persist_recovery_complete(p))
|
||||
return 0;
|
||||
|
||||
/* Logical entity restore only. persist_wal_write_replicated (above) already
|
||||
* wrote the entry and owns the on-disk CRC chain + current_lsn — so we pass
|
||||
* the entry fields directly rather than fabricating a header. (bug-778:
|
||||
* a synthetic crc=0 header here used to clobber p->last_crc.) */
|
||||
return apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
|
||||
e->raw_payload, (uint32_t)e->raw_payload_len);
|
||||
/* commit-then-apply on the standby (2026-07-10): entity materialization
|
||||
* (restore_job/restore_job_state) does NOT happen here on receipt. This
|
||||
* function's job ends at durable persist + ACK — the entry becomes
|
||||
* in-memory state only once this node's OWN p->committed_lsn reaches it,
|
||||
* driven by ha_standby_advance_committed_and_apply (heartbeat-learned
|
||||
* commit position, clamped to this node's local durable tip) and applied
|
||||
* through apply_committed_entry() by the same pump the primary's own
|
||||
* online-apply and restart replay use. Applying eagerly here — before
|
||||
* this node (or the sender) knows the entry reached cluster majority —
|
||||
* could materialize a job the sender never actually committed: if a
|
||||
* later (term,lsn) conflict truncate removes the WAL bytes (the sender
|
||||
* lost leadership before reaching majority), nothing rolls back the
|
||||
* already-applied entity mutation, leaving a permanent phantom job with
|
||||
* no backing WAL entry (2026-07-10 controller-invariant audit finding).
|
||||
* This mirrors bug-1043's own precedent of decoupling durability/ACK from
|
||||
* entity materialization, extended one step further: apply now waits for
|
||||
* COMMIT, not just local durability. */
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ha_send_wal_ack_reply: send WalAckReply {term, lsn, success=1} back to
|
||||
|
|
@ -2329,32 +2512,18 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
|
|||
* logOk and stalling commit. */
|
||||
(void)before_term;
|
||||
|
||||
/* bug-1043: the library route (on_persist_entry → persist_wal_write_
|
||||
* replicated) writes the entry to the WAL but does NOT apply it to the
|
||||
* in-memory entity state (job/queue/worker tables). The legacy direct-
|
||||
* apply path (ha_apply_single_entry) called apply_wal_entry after the
|
||||
* persist; the term-safe library route dropped it. So a standby that
|
||||
* caught up entries 8..N via replication held them on disk but NOT in the
|
||||
* queryable job table — on promotion `joblist`/`expect-job` could not
|
||||
* find a committed job (it IS on disk; walcompare passes), looking like a
|
||||
* committed-job loss. Re-apply here, mirroring the legacy path. Idempotent
|
||||
* on a re-stream (restore_job upserts by jobid). WAL_NOOP / WAL_CONFIG_
|
||||
* MARKER carry no entity payload — apply_wal_entry no-ops on them.
|
||||
*
|
||||
* bug-1416 (raft-apply-decoupling task 7.2 round 2): gated on
|
||||
* persist_recovery_complete() for the identical reason documented on
|
||||
* ha_apply_single_entry's own matching guard above — defer the
|
||||
* in-memory apply while this node's own boot-time persist_recover()
|
||||
* pass is in progress, never the ACK/applied_lsn advance below (the
|
||||
* entry is already durably persisted by on_persist_entry's
|
||||
* persist_wal_write_replicated, invoked via ha_consume_append_entries
|
||||
* above). This node's own in-progress replay, or the deferred
|
||||
* catch-up kick persist_recover() issues before returning, applies
|
||||
* it instead. */
|
||||
if (persist_recovery_complete(p))
|
||||
apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
|
||||
e->raw_payload, (uint32_t)e->raw_payload_len);
|
||||
|
||||
/* commit-then-apply on the standby (2026-07-10, supersedes the prior
|
||||
* bug-1043 re-apply-here fix): entity materialization does NOT happen
|
||||
* on receipt/persist. bug-1043's original problem — a caught-up
|
||||
* standby held entries on disk but not in the queryable job table —
|
||||
* is now solved by ha_standby_advance_committed_and_apply, driven by
|
||||
* this node's heartbeat-learned commit position, applying through the
|
||||
* SAME apply_committed_entry() seam the primary's online-apply and
|
||||
* restart replay use. Applying here instead (right on receipt, before
|
||||
* this entry is known to have reached cluster majority) risks a
|
||||
* permanent phantom job: a later (term,lsn) conflict truncate removes
|
||||
* the WAL bytes with no entity rollback (2026-07-10 controller-
|
||||
* invariant audit finding). */
|
||||
ha_advance_applied_lsn(ha, e->lsn);
|
||||
ha_send_wal_ack_reply(ha, ep, c, e->lsn);
|
||||
return 1;
|
||||
|
|
@ -2386,6 +2555,13 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
|
|||
if (c && c->ha.raft_c == c && entry->n_entries == 1 &&
|
||||
entry->entries && entry->entries[0] && entry->entries[0]->term != 0) {
|
||||
ha_apply_term_entry_via_library(ha, p, c, ep, entry->entries[0]);
|
||||
/* 2026-07-10: a fresh entry just landed on disk -- kick the apply
|
||||
* pump against the already-known commit position so a standby mid
|
||||
* catch-up doesn't wait for the next heartbeat tick to materialize
|
||||
* entries it's already known to have committed. */
|
||||
ha_standby_advance_committed_and_apply(
|
||||
c, atomic_load_explicit(&ha->last_known_committed_lsn,
|
||||
memory_order_acquire));
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -2461,6 +2637,15 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
|
|||
if (had_good)
|
||||
ha_advance_applied_lsn(ha, last_good_lsn);
|
||||
|
||||
/* 2026-07-10: kick the apply pump against the already-known commit
|
||||
* position so a batch catch-up materializes promptly instead of
|
||||
* waiting for the next heartbeat tick (see the single-entry-term-safe
|
||||
* branch above for the full rationale). */
|
||||
if (had_good)
|
||||
ha_standby_advance_committed_and_apply(
|
||||
c, atomic_load_explicit(&ha->last_known_committed_lsn,
|
||||
memory_order_acquire));
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -2499,6 +2684,12 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
|
|||
|
||||
/* Task 3.1: ACK the successfully applied single entry */
|
||||
ha_send_wal_ack_reply(ha, ep, c, entry->lsn);
|
||||
|
||||
/* 2026-07-10: kick the apply pump against the already-known commit
|
||||
* position (see the batch-mode branch above for the full rationale). */
|
||||
ha_standby_advance_committed_and_apply(
|
||||
c, atomic_load_explicit(&ha->last_known_committed_lsn,
|
||||
memory_order_acquire));
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -3560,6 +3751,7 @@ int ha_dispatch_gate(ha_state_t *ha, int opcode)
|
|||
if (opcode == OPCode_requestVote ||
|
||||
opcode == OPCode_preVoteRequest ||
|
||||
opcode == OPCode_probeConfigTag ||
|
||||
opcode == OPCode_heartbeat ||
|
||||
opcode == OPCode_hello ||
|
||||
opcode == OPCode_nop ||
|
||||
opcode == OPCode_replConfigPush ||
|
||||
|
|
@ -6806,6 +6998,28 @@ int ha_prevote_round_timeout_fn(void *arg)
|
|||
{
|
||||
prevote_round_ctx_t *ctx = (prevote_round_ctx_t *)arg;
|
||||
|
||||
/* bug (2026-07-10 chaotic-soak ASAN finding): clear s_prevote_round_ctx
|
||||
* for THIS ctx BEFORE freeing it, unconditionally -- regardless of which
|
||||
* branch below runs. The prior code only cleared the static pointer
|
||||
* inside the "round still in progress" branch; when pre_vote_in_progress
|
||||
* was ALREADY false here (the round was decided via a reply arriving on
|
||||
* a different path, one that doesn't itself clear this pointer) this
|
||||
* function fell straight through to free(ctx) below with
|
||||
* s_prevote_round_ctx still pointing at the about-to-be-freed memory.
|
||||
* The NEXT PreVote round's ha_start_prevote_async then calls
|
||||
* ha_prevote_cancel_round_timeout() defensively before arming its own
|
||||
* timer, reads that dangling pointer, and writes through it --
|
||||
* heap-use-after-free (confirmed via ASAN on a live chaotic HA soak,
|
||||
* both crashing nodes wedged with an identical ha.c:6712 UAF). CAS so a
|
||||
* newer round's context (if one has already replaced this one by the
|
||||
* time this delaymgr task runs) is never incorrectly cleared. */
|
||||
{
|
||||
prevote_round_ctx_t *expected = ctx;
|
||||
atomic_compare_exchange_strong_explicit(
|
||||
&s_prevote_round_ctx, &expected, NULL,
|
||||
memory_order_acq_rel, memory_order_relaxed);
|
||||
}
|
||||
|
||||
if (atomic_load_explicit(&ctx->cancelled, memory_order_acquire)) {
|
||||
#ifndef XCE_TESTING
|
||||
free(ctx);
|
||||
|
|
@ -6819,7 +7033,6 @@ int ha_prevote_round_timeout_fn(void *arg)
|
|||
/* Only act if the round is still in progress (not already decided). */
|
||||
if (ha->raft.pre_vote_in_progress) {
|
||||
ha->raft.pre_vote_in_progress = 0;
|
||||
atomic_store_explicit(&s_prevote_round_ctx, NULL, memory_order_release);
|
||||
WARN("HA: PreVote round timed out before a decision — treating as LOSS, "
|
||||
"staying CTRL_STANDBY; re-arming liveness watchdog (Req 5.4)");
|
||||
ha_prevote_rearm_liveness(ha, c);
|
||||
|
|
@ -7335,6 +7548,61 @@ typedef struct {
|
|||
*/
|
||||
static liveness_poll_ctx_t * _Atomic s_liveness_ctx = NULL;
|
||||
|
||||
/* rdhup_failover_vtask_entry / ha_dispatch_rdhup_failover: offload
|
||||
* ha_on_primary_ctrl_fd_rdhup (which chains into ha_start_election_async ->
|
||||
* raft_start_election -> on_persist -> ha_persist_term's synchronous fsync)
|
||||
* off the delaymgr thread onto a VTASK_LIVE_BLOCK vtask, mirroring
|
||||
* ha_dispatch_election_start. This is the real, common production trigger for
|
||||
* a standby's failover election (heartbeat-timeout watchdog) -- delaymgr must
|
||||
* only detect the timeout and hand off, never run the election itself. */
|
||||
typedef struct {
|
||||
ha_state_t *ha;
|
||||
cluster_t *c;
|
||||
} rdhup_failover_dispatch_t;
|
||||
|
||||
static int rdhup_failover_vtask_entry(vtask_t *task)
|
||||
{
|
||||
rdhup_failover_dispatch_t *d =
|
||||
(rdhup_failover_dispatch_t *)task->data->u.generic;
|
||||
ha_on_primary_ctrl_fd_rdhup(d->ha, d->c);
|
||||
free(d);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void ha_dispatch_rdhup_failover(ha_state_t *ha, cluster_t *c)
|
||||
{
|
||||
vtaskmgr_t *pool = (vtaskmgr_t *)ha->pool;
|
||||
vtask_data_t *data = calloc(1, sizeof(*data));
|
||||
vtask_t *task = vtask_alloc();
|
||||
rdhup_failover_dispatch_t *d = calloc(1, sizeof(*d));
|
||||
|
||||
if (pool == NULL || data == NULL || task == NULL || d == NULL) {
|
||||
WARN("ha_dispatch_rdhup_failover: vtask unavailable (pool=%p data=%p "
|
||||
"task=%p d=%p) — failing over inline (stay alive)",
|
||||
(void *)pool, (void *)data, (void *)task, (void *)d);
|
||||
free(data);
|
||||
free(task);
|
||||
free(d);
|
||||
ha_on_primary_ctrl_fd_rdhup(ha, c);
|
||||
return;
|
||||
}
|
||||
|
||||
d->ha = ha;
|
||||
d->c = c;
|
||||
data->vtype = VTASK_GENERIC;
|
||||
data->u.generic = d;
|
||||
vtask_init(task, (vtask_func_t)rdhup_failover_vtask_entry, data);
|
||||
|
||||
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
|
||||
WARN("ha_dispatch_rdhup_failover: vtask enqueue failed — failing over "
|
||||
"inline (stay alive)");
|
||||
free(data);
|
||||
free(task);
|
||||
free(d);
|
||||
ha_on_primary_ctrl_fd_rdhup(ha, c);
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* liveness_poll_fn: delaymgr task replacing hb_timeout_thread.
|
||||
*
|
||||
|
|
@ -7384,7 +7652,7 @@ static int liveness_poll_fn(void *arg)
|
|||
/* Silence exceeded timeout — trigger failover */
|
||||
WARN("HA: liveness poll: no primary activity for %lus (timeout=%us) → failover",
|
||||
(unsigned long)elapsed, timeout_s);
|
||||
ha_on_primary_ctrl_fd_rdhup(ha, c);
|
||||
ha_dispatch_rdhup_failover(ha, c);
|
||||
/* Do NOT re-arm — failover handles re-election */
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -7555,6 +7823,56 @@ void ha_observe_heartbeat_committed(cluster_t *c, uint64_t committed_lsn,
|
|||
committed_lsn, committed_config_tag, cur);
|
||||
}
|
||||
|
||||
/*
|
||||
* ha_standby_advance_committed_and_apply (2026-07-10, controller-invariant
|
||||
* audit finding — commit/apply decoupling extended to the standby): drive
|
||||
* this node's OWN p->committed_lsn from a learned primary committed position,
|
||||
* clamped to what this node has actually durably received, then kick the
|
||||
* SAME apply pump (persist_apply_pump_kick -> apply_committed_entry()) the
|
||||
* primary's own online-apply and restart replay use.
|
||||
*
|
||||
* This is the missing half of the standby ingest path's commit-then-apply
|
||||
* story: ha_apply_single_entry / ha_apply_term_entry_via_library persist +
|
||||
* ACK a replicated entry unconditionally (required for durability and the
|
||||
* primary's commit majority) but no longer apply it — entity materialization
|
||||
* now waits for THIS function to observe that the entry is actually
|
||||
* committed, never merely durable-on-receipt. Safe against a lagging
|
||||
* standby: the clamp to local_tip means committed_lsn here can never run
|
||||
* ahead of what this node has itself received, so persist_apply_pump_run's
|
||||
* apply_pump_read_entry can never be asked to read an LSN this node doesn't
|
||||
* have on disk yet.
|
||||
*
|
||||
* Race-safe against a CONCURRENT writer (2026-07-10, controller-invariant-
|
||||
* audit finding): this function is gated to non-primary roles by its
|
||||
* callers, but raft_cb_become_leader's own persist_promotion_advance_committed
|
||||
* call happens BEFORE role flips to CTRL_PRIMARY (ha.c, the role store sits
|
||||
* ~170 lines after the advance call) -- so a still-in-flight WAL catch-up
|
||||
* entry (this function is also called from ha_apply_wal_catchup_entry's exit
|
||||
* points) can legitimately run on another thread during that window, racing
|
||||
* this function against the promotion path's own advance. Both now go
|
||||
* through persist_committed_lsn_advance_atomic's CAS loop, which is safe
|
||||
* against that race by construction (never loses an update, never
|
||||
* regresses) -- no shared lock needed across the two call sites.
|
||||
*/
|
||||
static void ha_standby_advance_committed_and_apply(cluster_t *c,
|
||||
uint64_t learned_committed_lsn)
|
||||
{
|
||||
if (!c || learned_committed_lsn == 0)
|
||||
return;
|
||||
|
||||
persist_t *p = &c->persist;
|
||||
|
||||
pthread_mutex_lock(&p->lock);
|
||||
uint64_t local_tip = p->current_lsn > 0 ? p->current_lsn - 1 : 0;
|
||||
pthread_mutex_unlock(&p->lock);
|
||||
|
||||
uint64_t target = learned_committed_lsn < local_tip
|
||||
? learned_committed_lsn : local_tip;
|
||||
|
||||
persist_committed_lsn_advance_atomic(p, target);
|
||||
persist_apply_pump_kick(p, c->ha.pool);
|
||||
}
|
||||
|
||||
/*
|
||||
* ha_consume_heartbeat_committed: standby OP_nop consume branch — the
|
||||
* non-primary gate (Req 3.4) + the stale-term guard (Req 3.2), then the
|
||||
|
|
@ -7612,6 +7930,11 @@ void ha_consume_heartbeat_committed(cluster_t *c, const heartbeat_msg_t *hb)
|
|||
* non-advancing LSN no-ops inside the observe (Req 4.2). */
|
||||
ha_observe_heartbeat_committed(c, hb->committed_lsn,
|
||||
hb->committed_config_tag);
|
||||
|
||||
/* 2026-07-10: drive this node's own committed_lsn + apply pump from the
|
||||
* position just learned (see ha_standby_advance_committed_and_apply) —
|
||||
* the commit-then-apply half of the standby ingest path. */
|
||||
ha_standby_advance_committed_and_apply(c, hb->committed_lsn);
|
||||
}
|
||||
|
||||
void ha_disarm_heartbeat_timeout_vtask(ha_state_t *ha)
|
||||
|
|
@ -8326,6 +8649,28 @@ void ha_start_primary_heartbeat(cluster_t *c)
|
|||
return;
|
||||
}
|
||||
|
||||
/* 2026-07-10 (chaotic-soak finding): fire the FIRST heartbeat synchronously,
|
||||
* right now, instead of only arming the periodic delay_task_ms below.
|
||||
* delay_task_ms's first invocation fires only AFTER interval_ms elapses
|
||||
* (it is a periodic timer, not "run now then repeat") -- so despite this
|
||||
* function's own long-standing comment/doc ("start primary heartbeat
|
||||
* immediately so standbys receive NOP"), a freshly-promoted leader was
|
||||
* actually waiting a full heartbeat_interval_ms (default 3000ms) before a
|
||||
* reconnected/catching-up standby ever got a heartbeat to reply to. That
|
||||
* reply (stateReport) is what lets the primary learn the standby's true
|
||||
* applied_lsn (ha_process_standby_state) and start catch-up streaming --
|
||||
* which gates the per-entry majority-ack that advances committed_lsn past
|
||||
* this election's conservative safe_committed_lsn floor. A live chaotic
|
||||
* soak showed jobs above safe_lsn (legitimately majority-committed before
|
||||
* the old primary died, per Leader Completeness) invisible via
|
||||
* expect-job/joblist for the full 3s wait even after the
|
||||
* persist_promotion_advance_committed fix above -- because nothing
|
||||
* corroborated them was ever ASKED for that long. primary_heartbeat_fn
|
||||
* only performs non-blocking async sends (xce_ha_heartbeat_req), so
|
||||
* calling it inline here carries the same non-blocking-I/O contract as
|
||||
* the rest of promotion and needs no vtask/thread-role change. */
|
||||
primary_heartbeat_fn(c);
|
||||
|
||||
/* runcount=-1 (infinite): delaymgr re-enqueues automatically.
|
||||
* The role check in primary_heartbeat_fn no-ops on demotion. */
|
||||
int rc = delay_task_ms(&c->delaymgr, interval_ms, -1,
|
||||
|
|
|
|||
|
|
@ -227,6 +227,25 @@ typedef struct {
|
|||
* once per edge (no log storms across
|
||||
* repeated dispatch-gate rejections).
|
||||
* 0 at init (cleared by ha_init memset). */
|
||||
/* ── New field (2026-07-10, ha-convergence-recovery Req 1.4 supersession) ── */
|
||||
_Atomic int self_restart_pending; /* Set when ha_self_restart's execv()
|
||||
* fails (a demoted ex-primary could not
|
||||
* re-exec to discard provisional state).
|
||||
* Checked unconditionally, alongside
|
||||
* apply_divergent, by raft_cb_become_
|
||||
* leader and promotion_fence_timeout_fn
|
||||
* -- NEVER live-cleared (no clear site
|
||||
* exists anywhere): a periodic re-check
|
||||
* like promotion_fence_timeout_fn's
|
||||
* catch-up clear must NOT lift this, since
|
||||
* this node's WAL/durability state can
|
||||
* look perfectly healthy while its
|
||||
* in-memory state is still the stale
|
||||
* pre-demotion primary-era state the
|
||||
* restart exists to discard. Clears only
|
||||
* via a fresh process (a successful
|
||||
* retry replaces the image entirely).
|
||||
* 0 at init (cleared by ha_init memset). */
|
||||
/* ── New field added by ha-commit-durability spec (Task 4.4) ──────── */
|
||||
void * _Atomic promotion_fence_ctx; /* promotion_fence_ctx_t* for the
|
||||
* bounded-window catch-up timer (Req
|
||||
|
|
|
|||
|
|
@ -65,19 +65,73 @@ void (*ha_quorum_transition_log_hook)(int lost, int connected,
|
|||
int required) = NULL;
|
||||
#endif
|
||||
|
||||
#ifndef XCE_TESTING
|
||||
#define HA_SELF_RESTART_RETRY_MS 5000
|
||||
|
||||
static int ha_self_restart_retry_fn(void *arg);
|
||||
|
||||
/* ha_self_restart_enter_fence_and_retry: the non-abort response to a failed
|
||||
* re-exec (2026-07-10, supersedes ha-convergence-recovery Req 1.4's original
|
||||
* "terminate fatally" text — see requirements.md's superseding note). Sets
|
||||
* the sticky, never-live-cleared ha->self_restart_pending flag (checked
|
||||
* unconditionally by raft_cb_become_leader / promotion_fence_timeout_fn
|
||||
* alongside apply_divergent, ha.c) so this node can never become primary
|
||||
* while it still carries its stale pre-demotion in-memory state, then
|
||||
* schedules a bounded periodic retry of the re-exec. This satisfies the same
|
||||
* safety property Req 1.4 protects — a divergent ex-primary never serves
|
||||
* authoritatively — via the project-wide fail-loud invariant (a controller
|
||||
* MUST NEVER self-terminate as a designed response to an operational
|
||||
* condition; CLAUDE.md "Process Lifecycle") instead of abort(): staying
|
||||
* alive and retrying self-heals automatically once the transient condition
|
||||
* (ENOMEM, EMFILE, a permissions change mid-upgrade) clears, whereas a dead
|
||||
* process needs manual admin recovery and degrades quorum meanwhile. */
|
||||
static void ha_self_restart_enter_fence_and_retry(cluster_t *c)
|
||||
{
|
||||
atomic_store_explicit(&c->ha.self_restart_pending, 1, memory_order_release);
|
||||
|
||||
if (c->delaymgr.running) {
|
||||
int rc = delay_task_ms(&c->delaymgr, HA_SELF_RESTART_RETRY_MS, 1,
|
||||
ha_self_restart_retry_fn, c);
|
||||
if (rc < 0)
|
||||
WARN("HA self-restart: retry scheduling failed: %d — node stays "
|
||||
"fenced (non-promotable) with no further automatic retry "
|
||||
"attempt", rc);
|
||||
} else {
|
||||
WARN("HA self-restart: delaymgr not running — node stays fenced "
|
||||
"(non-promotable) with no automatic retry");
|
||||
}
|
||||
}
|
||||
|
||||
/* ha_self_restart_retry_fn: delaymgr task body — bounded periodic retry of
|
||||
* the re-exec. A successful retry replaces the process image entirely (this
|
||||
* function never returns in that case); a repeated failure re-arms itself
|
||||
* via ha_self_restart's own failure path. */
|
||||
static int ha_self_restart_retry_fn(void *arg)
|
||||
{
|
||||
cluster_t *c = (cluster_t *)arg;
|
||||
if (!c)
|
||||
return -EINVAL;
|
||||
return ha_self_restart(c);
|
||||
}
|
||||
#endif
|
||||
|
||||
/*
|
||||
* ha_self_restart — the truncate-and-restart primitive's restart half.
|
||||
*
|
||||
* Production (Req 1.3, 1.4): flush logs, then execv() the saved argv so the
|
||||
* Production (Req 1.3): flush logs, then execv() the saved argv so the
|
||||
* controller rebuilds in-memory state purely from the surviving (committed)
|
||||
* WAL prefix via boot recovery. If exec cannot be performed, terminate
|
||||
* FATALLY (abort) — a demoted ex-primary must NEVER continue running with
|
||||
* potentially divergent state. NEVER returns in production.
|
||||
* WAL prefix via boot recovery. NEVER returns on success (execv replaces the
|
||||
* process image). If exec cannot be performed, do NOT abort (see
|
||||
* ha_self_restart_enter_fence_and_retry above) — enter a permanent
|
||||
* non-promotable fence and keep retrying periodically, per the project's
|
||||
* never-self-terminate invariant.
|
||||
*
|
||||
* Test (Req 1.6): under XCE_TESTING invoke ha_restart_hook(c) instead of
|
||||
* execv and return its result on the success path; a non-zero hook return
|
||||
* models an exec failure and routes to ha_fatal_hook in place of abort, so a
|
||||
* test binary can assert the FATAL path was taken without dying (Req 1.4).
|
||||
* models an exec failure and routes to ha_fatal_hook (retained as the test
|
||||
* seam name — it still marks "restart could not be performed", now backed by
|
||||
* the fence-and-retry response in production rather than abort), so a test
|
||||
* binary can assert that failure path was taken without dying.
|
||||
*/
|
||||
int ha_self_restart(cluster_t *c)
|
||||
{
|
||||
|
|
@ -88,20 +142,26 @@ int ha_self_restart(cluster_t *c)
|
|||
rc = ha_restart_hook(c);
|
||||
|
||||
if (rc != 0 || ha_restart_hook == NULL) {
|
||||
/* Restart could not be performed → FATAL contract (Req 1.4). */
|
||||
/* Restart could not be performed — same non-abort response as
|
||||
* production (Req 1.6 mirrors ha_self_restart_enter_fence_and_retry's
|
||||
* flag-setting without the real delaymgr retry, since a unit fixture
|
||||
* has no live delaymgr to retry on). */
|
||||
if (c)
|
||||
atomic_store_explicit(&c->ha.self_restart_pending, 1,
|
||||
memory_order_release);
|
||||
if (ha_fatal_hook != NULL)
|
||||
ha_fatal_hook("ha_self_restart: restart seam failed (FATAL)");
|
||||
ha_fatal_hook("ha_self_restart: restart seam failed — fenced "
|
||||
"non-promotable, staying alive");
|
||||
return rc != 0 ? rc : -1;
|
||||
}
|
||||
return 0;
|
||||
#else
|
||||
(void)c;
|
||||
|
||||
char **argv = controller_saved_argv();
|
||||
if (argv == NULL || argv[0] == NULL) {
|
||||
ERROR("HA self-restart: no saved argv — cannot re-exec (FATAL)");
|
||||
tlog_exit(); /* flush pending log buffer before dying */
|
||||
abort();
|
||||
ERROR("HA self-restart: no saved argv — cannot re-exec; entering "
|
||||
"non-promotable fence and retrying periodically (stay alive)");
|
||||
ha_self_restart_enter_fence_and_retry(c);
|
||||
return -1;
|
||||
}
|
||||
|
||||
INFO("HA self-restart: re-executing %s to rebuild from committed state",
|
||||
|
|
@ -110,12 +170,16 @@ int ha_self_restart(cluster_t *c)
|
|||
|
||||
execv(argv[0], argv);
|
||||
|
||||
/* execv only returns on failure — FATAL (Req 1.4): never continue serving
|
||||
* as a divergent ex-primary. */
|
||||
ERROR("HA self-restart: execv(%s) failed: %d — aborting (FATAL)",
|
||||
argv[0], errno);
|
||||
abort();
|
||||
return -1; /* unreachable */
|
||||
/* execv only returns on failure. A demoted ex-primary must never continue
|
||||
* serving with potentially divergent state (ha-convergence-recovery Req
|
||||
* 1.4) — but per the project-wide fail-loud invariant (never self-
|
||||
* terminate as a designed response to an operational condition), the
|
||||
* correct response is a permanent non-promotable fence that stays alive
|
||||
* and keeps retrying, not abort(). */
|
||||
ERROR("HA self-restart: execv(%s) failed: %d — entering non-promotable "
|
||||
"fence and retrying periodically (stay alive)", argv[0], errno);
|
||||
ha_self_restart_enter_fence_and_retry(c);
|
||||
return -1;
|
||||
#endif
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@
|
|||
#include <unistd.h>
|
||||
|
||||
#include "cluster.h"
|
||||
#include "config_repl.h"
|
||||
#include "ha.h"
|
||||
#include "persist.h"
|
||||
#include "xce/cluster_def.h"
|
||||
|
|
@ -172,40 +171,36 @@ static int handle_scheduler_join(cluster_t* c, endpoint_t* ep,
|
|||
return rc;
|
||||
}
|
||||
|
||||
/*
|
||||
* handle_controller_join: reject a peer-controller join request.
|
||||
*
|
||||
* Controller-to-controller connection establishment is NOT this path — per
|
||||
* ha-async-io.md's Connection Ownership, it is always primary-initiated
|
||||
* outbound (primary_heartbeat_fn -> on_standby_connected -> ctrlhello ->
|
||||
* repl_add_standby), with the resulting endpoint owned by
|
||||
* vote_ep_err_handler / standby_ep_err_handler_primary. This arm used to
|
||||
* accept an INBOUND role=HostRole_controller join and store it directly
|
||||
* into standby_endpoints, bypassing both documented owners and their
|
||||
* err-handler-driven eviction — a live ownership-model violation, even
|
||||
* though no current sender in the tree ever builds such a request
|
||||
* (xce_join_req structurally rejects role==HostRole_controller). Reject
|
||||
* outright rather than leave the bypass reachable from the wire.
|
||||
*/
|
||||
static int handle_controller_join(cluster_t* c, endpoint_t* ep,
|
||||
xce_request_t* request)
|
||||
{
|
||||
xce_reply_t reply;
|
||||
|
||||
init_reply_by_req(&reply, request);
|
||||
reply.status = 0;
|
||||
reply.status = -E_XCE_BAD_ROLE;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
|
||||
/* Register this endpoint for OP_upgradeConfig fan-out to standbys */
|
||||
if (c->ha.standby_endpoints) {
|
||||
ep_private_t *priv = (ep_private_t *)ep->data;
|
||||
if (priv)
|
||||
priv->role = HostRole_controller;
|
||||
WARN("%s rejecting controller-role join from %s — controller peering is "
|
||||
"primary-initiated only, never via join",
|
||||
__func__, request->from ? request->from->name : "?");
|
||||
|
||||
pthread_mutex_lock(&c->ha.lock);
|
||||
zarray_store(c->ha.standby_endpoints, ep);
|
||||
pthread_mutex_unlock(&c->ha.lock);
|
||||
|
||||
INFO("controller %s joined — %d standby(s) tracked",
|
||||
request->from ? request->from->name : "?",
|
||||
c->ha.standby_endpoints->count);
|
||||
|
||||
/* Immediately push current config to the newly-joined standby so it
|
||||
* doesn't have to wait for the next scheduled push. Transport-only
|
||||
* (no barrier wait — task 5.2 remediation r1): this is a best-effort
|
||||
* catch-up stream; convergence is owned by the OP_replConfig
|
||||
* anti-entropy pull and the heartbeat decision loop. */
|
||||
if (ha_is_primary(&c->ha))
|
||||
push_config_transport_only(c, NULL, 0, NULL, 0, NULL);
|
||||
}
|
||||
|
||||
return 0;
|
||||
return -E_XCE_BAD_ROLE;
|
||||
}
|
||||
|
||||
static int handle_apiserver_join(cluster_t* c, endpoint_t* ep,
|
||||
|
|
|
|||
|
|
@ -3441,6 +3441,55 @@ void persist_apply_pump_kick(persist_t* p, void* pool_arg)
|
|||
}
|
||||
}
|
||||
|
||||
/* persist_apply_pump_run_sync (2026-07-10, chaotic-soak regression fix,
|
||||
* ROUND 2): synchronously drain the apply pump on THIS thread, GUARANTEEING
|
||||
* last_applied_lsn reaches committed_lsn (as observed at entry) before
|
||||
* returning -- regardless of what any concurrent caller is doing.
|
||||
*
|
||||
* Round 1 of this fix CAS-claimed apply_pump_scheduled and, on losing that
|
||||
* claim to a concurrent caller (e.g. a heartbeat-driven kick that fired
|
||||
* moments before promotion), just returned -- betting the OTHER caller's
|
||||
* async vtask-dispatched drain would finish in time. A second live chaotic
|
||||
* soak proved that bet wrong: job loss persisted even with the environmental
|
||||
* fdatasync noise (bug-924) independently confirmed and eliminated, so this
|
||||
* was a genuine, separate bug in the fix itself, not noise. The caller here
|
||||
* (promotion) cannot outsource its guarantee to a drain it doesn't control
|
||||
* the completion of.
|
||||
*
|
||||
* Fix: do NOT use the CAS-coalescing flag as a "someone else has it, I'm
|
||||
* done" signal -- call persist_apply_pump_run() directly in a loop until
|
||||
* THIS call observes last_applied_lsn caught up to committed_lsn. Multiple
|
||||
* concurrent callers (this one and any vtask-dispatched persist_apply_pump_
|
||||
* kick) safely interleave via persist_apply_pump_run's own internal
|
||||
* apply_pump_lock and apply_committed_entry's per-LSN idempotency -- at
|
||||
* worst a redundant no-op pass, never lost or duplicated work. Stops (does
|
||||
* not busy-loop forever) on a genuine read failure, exactly mirroring
|
||||
* apply_pump_drain's own error handling: persist_apply_pump_run() already
|
||||
* logs loudly and leaves last_applied_lsn unadvanced on that path. */
|
||||
void persist_apply_pump_run_sync(persist_t* p)
|
||||
{
|
||||
if (!p || !p->cluster)
|
||||
return;
|
||||
if (!atomic_load_explicit(&p->recovery_complete, memory_order_acquire))
|
||||
return;
|
||||
|
||||
for (;;) {
|
||||
uint64_t committed = atomic_load_explicit(&p->committed_lsn,
|
||||
memory_order_acquire);
|
||||
uint64_t applied = atomic_load_explicit(&p->last_applied_lsn,
|
||||
memory_order_relaxed);
|
||||
if (applied >= committed)
|
||||
return; /* caught up -- by us or a concurrent drain */
|
||||
|
||||
int rc = persist_apply_pump_run(p);
|
||||
if (rc < 0)
|
||||
return; /* already logged loudly; do not busy-loop on the same
|
||||
* unresolved entry (mirrors apply_pump_drain) */
|
||||
/* Loop: re-check in case committed_lsn advanced further mid-walk, or
|
||||
* a concurrent drain made partial progress this pass didn't see. */
|
||||
}
|
||||
}
|
||||
|
||||
/* Read WAL segment files in LSN order; replay entries with lsn > from_lsn.
|
||||
*
|
||||
* Segment filename format: seg-<start_lsn_hex16>.wal
|
||||
|
|
@ -5312,6 +5361,81 @@ int persist_truncate_wal_above(persist_t* p, uint64_t cutoff_lsn)
|
|||
return rc;
|
||||
}
|
||||
|
||||
/* persist_committed_lsn_advance_atomic (2026-07-10, controller-invariant-audit
|
||||
* finding): race-safe monotonic-max advance of p->committed_lsn via a CAS
|
||||
* retry loop. Replaces the naive "load; if target>cur store" pattern, which
|
||||
* can lose an update if two writers race: persist_promotion_advance_committed
|
||||
* (below) and ha_standby_advance_committed_and_apply (ha.c) can both run
|
||||
* during the promotion window before role flips to CTRL_PRIMARY (a
|
||||
* still-in-flight WAL catch-up entry processed on another thread at the same
|
||||
* instant as this node's own election completing). No lock required -- CAS
|
||||
* only succeeds if the value hasn't changed since the read, so a losing
|
||||
* racer retries with the freshly-observed value and can never clobber a
|
||||
* larger value with a smaller one. */
|
||||
uint64_t persist_committed_lsn_advance_atomic(persist_t* p, uint64_t target)
|
||||
{
|
||||
uint64_t cur = atomic_load_explicit(&p->committed_lsn, memory_order_acquire);
|
||||
while (target > cur) {
|
||||
if (atomic_compare_exchange_weak_explicit(&p->committed_lsn, &cur, target,
|
||||
memory_order_release,
|
||||
memory_order_acquire))
|
||||
return target;
|
||||
/* cur was refreshed to the current value by the failed CAS; retry. */
|
||||
}
|
||||
return cur;
|
||||
}
|
||||
|
||||
/* persist_promotion_advance_committed (2026-07-10, promotion apply-lag fix):
|
||||
* unconditionally advance p->committed_lsn toward `safe_lsn` at promotion
|
||||
* time. persist_truncate_wal_above only sets committed_lsn on the branches
|
||||
* of truncate_wal_above_locked that perform a REAL truncation (partial
|
||||
* truncate, or the full-segment-purge demotion-convergence case); its no-op
|
||||
* early return (nothing above cutoff_lsn to remove -- the common case for a
|
||||
* newly-elected leader whose own log already covers safe_lsn) leaves
|
||||
* committed_lsn untouched at its pre-election value. Without this call,
|
||||
* committed_lsn -- and therefore apply, which is commit-gated
|
||||
* (raft-apply-decoupling) -- stalls until the normal per-entry ack-driven
|
||||
* commit path organically catches it up, which can take seconds while
|
||||
* surviving standbys reconnect and re-ACK. During that gap a job already
|
||||
* committed on a majority is invisible via expect-job/joblist on the very
|
||||
* primary that is authoritative for it (bug-828 class).
|
||||
*
|
||||
* DESIGN NOTE -- why this function never forces a synchronous flush (round 3,
|
||||
* 2026-07-10, controller-invariant-audit): an earlier round of this fix
|
||||
* called flusher_flush_once(p) here when durable_lsn lagged safe_lsn, to
|
||||
* force this node's own fsync to catch up immediately rather than waiting on
|
||||
* fsync_worker's cadence. That was a real bug-1198-CLASS thread-role
|
||||
* violation: the real multi-node winning-vote-reply path reaches
|
||||
* raft_cb_become_leader (and so this function) via peer_ctrl_classify's
|
||||
* VTASK_LIVENESS dispatch (ha.c ~5935-5940) -- a task class the design
|
||||
* DOCUMENTS as "never admission-gated" / never expected to block (see the
|
||||
* bug-1198 hazard box a few hundred lines above peer_ctrl_classify's
|
||||
* definition, which closed the EXACT same class of bug for
|
||||
* HA_REPL_SEND_SNAPSHOT's persist_checkpoint). A blocking fdatasync here
|
||||
* would reintroduce that closed bug class on the pool's liveness-reserved
|
||||
* workers. The visibility gap the force-flush was trying to close is instead
|
||||
* closed non-blockingly by ha_start_primary_heartbeat sending its first
|
||||
* heartbeat synchronously at promotion (bug-1432) -- that lets a reconnecting
|
||||
* standby's real position get learned, and the entry gets committed via the
|
||||
* normal ack-driven commit_advance_locked path, within one network RTT
|
||||
* instead of waiting a full heartbeat interval. A live chaotic soak (10
|
||||
* rounds) confirmed durable-capped-only + immediate-heartbeat together reach
|
||||
* committed_job_loss=0 with zero blocking calls added to this function.
|
||||
*
|
||||
* Durable-capped (never advances committed_lsn past this node's own
|
||||
* durable_lsn -- see persistence.md's "acknowledged implies durable on a
|
||||
* majority" core invariant) and monotonic + race-safe via
|
||||
* persist_committed_lsn_advance_atomic. */
|
||||
void persist_promotion_advance_committed(persist_t* p, uint64_t safe_lsn)
|
||||
{
|
||||
if (!p)
|
||||
return;
|
||||
uint64_t durable = atomic_load_explicit(&p->durable_lsn,
|
||||
memory_order_acquire);
|
||||
uint64_t target = safe_lsn < durable ? safe_lsn : durable;
|
||||
persist_committed_lsn_advance_atomic(p, target);
|
||||
}
|
||||
|
||||
/* Lock-held core of persist_truncate_wal_above (see the forward declaration and
|
||||
* the conflict-guard reuse near persist_wal_write_replicated). Caller MUST hold
|
||||
* p->lock; this function never touches the lock so it is reusable from another
|
||||
|
|
|
|||
|
|
@ -79,6 +79,7 @@
|
|||
#include "cluster.h"
|
||||
#include "commit.h"
|
||||
#include "controller.capnp.h"
|
||||
#include "ha.h"
|
||||
#include "hook.h"
|
||||
#include "jobj.h"
|
||||
#include "persist.h"
|
||||
|
|
@ -1058,6 +1059,235 @@ static void test_apply_pump_late_pending_apply_registration_self_heals(void)
|
|||
"PASS\n");
|
||||
}
|
||||
|
||||
/* ── 6. Standby-ingest commit-then-apply (2026-07-10 fix) ────────────────
|
||||
*
|
||||
* Regression coverage for ha.c's standby WAL-replication ingest path.
|
||||
* Before this fix, ha_apply_single_entry / ha_apply_term_entry_via_library
|
||||
* called apply_wal_entry() directly the instant a replicated entry was
|
||||
* durably persisted — materializing the job into the job table regardless
|
||||
* of whether the sending primary ever reached cluster-wide majority
|
||||
* commit. If that primary died before reaching majority, a later
|
||||
* (term,lsn) conflict truncate would remove the WAL bytes with no
|
||||
* rollback of the already-materialized entity, leaving a permanent
|
||||
* phantom job.
|
||||
*
|
||||
* The fix decouples durability from materialization: ha_apply_single_entry
|
||||
* / ha_apply_term_entry_via_library now only persist + ACK. Entity
|
||||
* materialization happens later, only once THIS standby's own
|
||||
* persist_t.committed_lsn actually covers the entry — driven by
|
||||
* ha_standby_advance_committed_and_apply (clamped to
|
||||
* min(learned_committed_lsn, local durable tip)) via
|
||||
* persist_apply_pump_kick -> persist_apply_pump_run -> apply_committed_entry
|
||||
* (the same seam the primary's own online-apply/replay use).
|
||||
*
|
||||
* These tests drive the REAL standby ingest seam — ha_apply_wal_catchup_entry,
|
||||
* the exact function every incoming WAL replication message routes through —
|
||||
* directly against this file's real jobmgr/persist fixture, so "not yet
|
||||
* visible" / "now visible" is observed via tap_find_job, the same job-table
|
||||
* lookup every other test in this file uses, not a synthetic stand-in.
|
||||
* g_tap_c.ha.raft_c is never set to &g_tap_c in this fixture, so
|
||||
* ha_apply_wal_catchup_entry always takes the documented term-blind
|
||||
* direct-persist bypass (ha_apply_single_entry) regardless of the entries'
|
||||
* term — the same reachable-and-safe-by-guard path
|
||||
* test_6_2_real_senders_never_produce_termblind_shape (test_ha_core_safety.c)
|
||||
* documents. */
|
||||
|
||||
/* Build one WAL_JOB_SUBMIT sub-entry for jobid at lsn, via the same capnp
|
||||
* encoding tap_encode_job already uses for the live command-entry test
|
||||
* above. Caller owns *payload_out (free() after the entry is no longer
|
||||
* needed) and must keep it alive across the ha_apply_wal_catchup_entry
|
||||
* call — persist_wal_write_replicated copies the bytes to disk, so the
|
||||
* pointer's lifetime need not outlive the call, but freeing before the
|
||||
* call would read stale data mid-encode. */
|
||||
static wal_catchup_entry_t tas_build_submit_entry(uint64_t lsn, uint64_t jobid,
|
||||
uint32_t state,
|
||||
uint8_t** payload_out)
|
||||
{
|
||||
memset(&g_tap_seed_submit, 0, sizeof(g_tap_seed_submit));
|
||||
g_tap_seed_submit.queue = (char*)"q";
|
||||
g_tap_seed_submit.user = (char*)"tester";
|
||||
|
||||
xce_job_t jdef;
|
||||
memset(&jdef, 0, sizeof(jdef));
|
||||
jdef.jobid = jobid;
|
||||
jdef.state = state;
|
||||
jdef.submit = &g_tap_seed_submit;
|
||||
|
||||
uint8_t* payload = NULL;
|
||||
size_t plen = 0;
|
||||
TEST_ASSERT_EQUAL_INT(0, tap_encode_job(&jdef, &payload, &plen));
|
||||
|
||||
wal_catchup_entry_t sub;
|
||||
memset(&sub, 0, sizeof(sub));
|
||||
sub.lsn = lsn;
|
||||
sub.entry_type = WAL_JOB_SUBMIT;
|
||||
sub.raw_payload = payload;
|
||||
sub.raw_payload_len = (uint32_t)plen;
|
||||
/* term == 0 deliberately: see the file-header comment above — this
|
||||
* fixture always exercises the term-blind direct-persist bypass. */
|
||||
|
||||
*payload_out = payload;
|
||||
return sub;
|
||||
}
|
||||
|
||||
/* (a) Positive case: persisted-but-uncommitted stays invisible; a
|
||||
* heartbeat-learned commit position materializes it. */
|
||||
static void test_standby_apply_deferred_until_committed_then_materializes(
|
||||
void)
|
||||
{
|
||||
tap_setup();
|
||||
|
||||
const uint64_t lsn = 1; /* first entry in a fresh persist_t */
|
||||
const uint64_t jobid = 960;
|
||||
|
||||
uint8_t* payload = NULL;
|
||||
wal_catchup_entry_t sub = tas_build_submit_entry(lsn, jobid, JOB_STAT_PEND,
|
||||
&payload);
|
||||
wal_catchup_entry_t* sub_arr[1] = { &sub };
|
||||
wal_catchup_entry_t top;
|
||||
memset(&top, 0, sizeof(top));
|
||||
top.n_entries = 1;
|
||||
top.entries = sub_arr;
|
||||
|
||||
/* Ingest via the REAL standby seam: persist + ACK only, no apply. */
|
||||
int rc = ha_apply_wal_catchup_entry(&g_tap_c.ha, &g_tap_c.persist, &top,
|
||||
NULL, &g_tap_c);
|
||||
|
||||
uint64_t current_lsn_after_persist = g_tap_c.persist.current_lsn;
|
||||
uint64_t applied_lsn_after_persist =
|
||||
atomic_load(&g_tap_c.ha.applied_lsn);
|
||||
uint64_t committed_after_persist = atomic_load(&g_tap_c.persist.committed_lsn);
|
||||
job_t* j_before_commit = tap_find_job(jobid);
|
||||
|
||||
/* Learn the commit position the way a real heartbeat does -- this is
|
||||
* what drives the commit-then-apply wiring (ha_consume_heartbeat_committed
|
||||
* -> ha_standby_advance_committed_and_apply -> persist_apply_pump_kick). */
|
||||
atomic_store(&g_tap_c.ha.role, (int32_t)CTRL_STANDBY);
|
||||
|
||||
heartbeat_msg_t hb;
|
||||
memset(&hb, 0, sizeof(hb));
|
||||
hb.term = 0;
|
||||
hb.committed_lsn = lsn;
|
||||
ha_consume_heartbeat_committed(&g_tap_c, &hb);
|
||||
|
||||
/* See tap_wait_applied's doc comment: the walk may complete on a
|
||||
* different thread than this call. */
|
||||
tap_wait_applied(lsn, 2000);
|
||||
|
||||
uint64_t committed_after_hb = atomic_load(&g_tap_c.persist.committed_lsn);
|
||||
job_t* j_after_commit = tap_find_job(jobid);
|
||||
uint32_t j_state_after = j_after_commit ? j_after_commit->job.state : 0;
|
||||
uint64_t last_applied_final = persist_last_applied_lsn(&g_tap_c.persist);
|
||||
|
||||
free(payload);
|
||||
tap_teardown();
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn + 1, current_lsn_after_persist,
|
||||
"the entry must be durably persisted by the standby ingest seam");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn, applied_lsn_after_persist,
|
||||
"applied_lsn (the held/persisted watermark) advances on receipt -- "
|
||||
"distinct from entity materialization");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, committed_after_persist,
|
||||
"committed_lsn must NOT advance just because the entry was "
|
||||
"persisted -- nothing has told this standby the entry reached "
|
||||
"cluster majority yet");
|
||||
TEST_ASSERT_NULL_MESSAGE(j_before_commit,
|
||||
"REGRESSION GUARD: a replicated entry must not materialize into "
|
||||
"the job table before this standby's own committed_lsn covers it "
|
||||
"-- materializing here (the old eager-apply behavior) is exactly "
|
||||
"the phantom-job hazard this fix closes");
|
||||
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn, committed_after_hb,
|
||||
"the heartbeat-learned commit position must advance committed_lsn");
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(j_after_commit,
|
||||
"once committed_lsn covers the entry, the apply pump must "
|
||||
"materialize it into the job table");
|
||||
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_PEND, j_state_after);
|
||||
TEST_ASSERT_EQUAL_UINT64(lsn, last_applied_final);
|
||||
|
||||
printf("test_standby_apply_deferred_until_committed_then_materializes: "
|
||||
"PASS\n");
|
||||
}
|
||||
|
||||
/* (b) Regression guard: an entry truncated away before ever being learned
|
||||
* as committed must never leave a phantom job — even when a stale/racing
|
||||
* heartbeat still advertises the old (now-truncated) committed position. */
|
||||
static void test_standby_truncate_before_commit_leaves_no_phantom_job(void)
|
||||
{
|
||||
tap_setup();
|
||||
|
||||
const uint64_t lsn = 1;
|
||||
const uint64_t jobid = 961;
|
||||
const uint64_t cutoff_lsn = lsn - 1; /* truncate away lsn and above */
|
||||
|
||||
uint8_t* payload = NULL;
|
||||
wal_catchup_entry_t sub = tas_build_submit_entry(lsn, jobid, JOB_STAT_PEND,
|
||||
&payload);
|
||||
wal_catchup_entry_t* sub_arr[1] = { &sub };
|
||||
wal_catchup_entry_t top;
|
||||
memset(&top, 0, sizeof(top));
|
||||
top.n_entries = 1;
|
||||
top.entries = sub_arr;
|
||||
|
||||
int rc = ha_apply_wal_catchup_entry(&g_tap_c.ha, &g_tap_c.persist, &top,
|
||||
NULL, &g_tap_c);
|
||||
|
||||
job_t* j_before_truncate = tap_find_job(jobid);
|
||||
|
||||
/* The sending primary died before reaching majority: a new leader's
|
||||
* conflict-truncate removes the entry before this standby ever learns
|
||||
* it was committed. */
|
||||
int trc = persist_truncate_wal_above(&g_tap_c.persist, cutoff_lsn);
|
||||
uint64_t current_lsn_after_truncate = g_tap_c.persist.current_lsn;
|
||||
|
||||
/* A stale/racing heartbeat still carries the old (now-truncated)
|
||||
* committed position -- ha_standby_advance_committed_and_apply's clamp
|
||||
* to the local durable tip (current_lsn-1) must refuse to walk past
|
||||
* what is actually on disk. */
|
||||
atomic_store(&g_tap_c.ha.role, (int32_t)CTRL_STANDBY);
|
||||
|
||||
heartbeat_msg_t hb;
|
||||
memset(&hb, 0, sizeof(hb));
|
||||
hb.term = 0;
|
||||
hb.committed_lsn = lsn;
|
||||
ha_consume_heartbeat_committed(&g_tap_c, &hb);
|
||||
|
||||
/* There is nothing valid left to apply, so no walk will ever reach
|
||||
* `lsn` -- a bounded settle (not a wait-for) is the correct shape here. */
|
||||
usleep(50000);
|
||||
|
||||
uint64_t committed_after = atomic_load(&g_tap_c.persist.committed_lsn);
|
||||
uint64_t last_applied_final = persist_last_applied_lsn(&g_tap_c.persist);
|
||||
job_t* j_after = tap_find_job(jobid);
|
||||
|
||||
free(payload);
|
||||
tap_teardown();
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_NULL_MESSAGE(j_before_truncate,
|
||||
"must not be materialized before commit (same guard as the "
|
||||
"positive-case test)");
|
||||
TEST_ASSERT_EQUAL_INT(0, trc);
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(cutoff_lsn + 1, current_lsn_after_truncate,
|
||||
"the entry must actually be gone from the WAL after truncation");
|
||||
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, committed_after,
|
||||
"REGRESSION GUARD: the local-durable-tip clamp in "
|
||||
"ha_standby_advance_committed_and_apply must refuse to advance "
|
||||
"committed_lsn past a truncated-away entry, even when a stale "
|
||||
"heartbeat still advertises it as committed");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_final,
|
||||
"nothing may be applied past the truncated tip");
|
||||
TEST_ASSERT_NULL_MESSAGE(j_after,
|
||||
"REGRESSION GUARD: no phantom job may appear -- the entry was "
|
||||
"truncated before ever being committed, so it must never be "
|
||||
"materialized");
|
||||
|
||||
printf("test_standby_truncate_before_commit_leaves_no_phantom_job: "
|
||||
"PASS\n");
|
||||
}
|
||||
|
||||
/* ── Test group entry point ───────────────────────────────────────────── */
|
||||
|
||||
void test_apply_pump(void)
|
||||
|
|
@ -1068,5 +1298,7 @@ void test_apply_pump(void)
|
|||
RUN_TEST(test_apply_pump_guard_does_not_break_job_ctrl_live_append);
|
||||
RUN_TEST(test_apply_pump_fail_racing_finish_merges_against_current_state);
|
||||
RUN_TEST(test_apply_pump_late_pending_apply_registration_self_heals);
|
||||
RUN_TEST(test_standby_apply_deferred_until_committed_then_materializes);
|
||||
RUN_TEST(test_standby_truncate_before_commit_leaves_no_phantom_job);
|
||||
UNITY_END();
|
||||
}
|
||||
|
|
|
|||
|
|
@ -964,6 +964,37 @@ int persist_checkpoint(persist_t* p);
|
|||
* Idempotent; returns 0 on success, negative errno on I/O error. */
|
||||
int persist_truncate_wal_above(persist_t* p, uint64_t cutoff_lsn);
|
||||
|
||||
/* persist_promotion_advance_committed (2026-07-10): called once by a newly-
|
||||
* promoted leader (raft_cb_become_leader) right after persist_truncate_wal_above,
|
||||
* to unconditionally advance committed_lsn toward the post-election safe commit
|
||||
* point `safe_lsn`. persist_truncate_wal_above only sets committed_lsn on a
|
||||
* REAL truncation; its common no-op path (this leader's own log already
|
||||
* covers safe_lsn) leaves committed_lsn stale, stalling commit-gated apply
|
||||
* until the normal ack-driven path catches up seconds later. Durable-capped
|
||||
* (never advances committed_lsn past this node's own durable_lsn) and
|
||||
* monotonic (never regresses) — uses persist_committed_lsn_advance_atomic, so
|
||||
* it is also race-safe against a concurrent advance from another writer (e.g.
|
||||
* ha_standby_advance_committed_and_apply, which can still run during the
|
||||
* promotion window before role flips to CTRL_PRIMARY). Does NOT force a
|
||||
* synchronous flush if durable_lsn lags — this function never blocks (see the
|
||||
* DESIGN NOTE at its definition for why a blocking fdatasync here would
|
||||
* violate the VTASK_LIVENESS thread-role contract, controller-invariant-audit
|
||||
* 2026-07-10, bug-1198-class finding). Safe to call with p == NULL (no-op). */
|
||||
void persist_promotion_advance_committed(persist_t* p, uint64_t safe_lsn);
|
||||
|
||||
/* persist_committed_lsn_advance_atomic: race-safe monotonic-max advance of
|
||||
* p->committed_lsn to max(current, target) via a CAS retry loop — safe
|
||||
* against ANY concurrent caller, no lock required or held. A naive
|
||||
* load-then-conditional-store (the pattern this replaces) can lose an update
|
||||
* under a genuine race: controller-invariant-audit (2026-07-10) found
|
||||
* persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
|
||||
* can race during the promotion window before role flips to CTRL_PRIMARY (a
|
||||
* still-in-flight WAL catch-up entry can be processed on another thread at
|
||||
* the same instant). Never regresses committed_lsn. Returns the value
|
||||
* observed to be current after the call (may exceed `target` if another
|
||||
* writer raced ahead — this is not a failure, just the up-to-date value). */
|
||||
uint64_t persist_committed_lsn_advance_atomic(persist_t* p, uint64_t target);
|
||||
|
||||
/* Logging helper — never logs payload content (security invariant) */
|
||||
void persist_log_entry_metadata(const wal_entry_hdr_t* hdr);
|
||||
|
||||
|
|
@ -1193,6 +1224,14 @@ int persist_apply_pump_run(persist_t* p);
|
|||
* most one walk is ever scheduled or running at a time per persist_t. */
|
||||
void persist_apply_pump_kick(persist_t* p, void* pool);
|
||||
|
||||
/* persist_apply_pump_run_sync (2026-07-10): synchronously drain the apply
|
||||
* pump on the calling thread instead of dispatching to a vtask. Only for
|
||||
* callers that can tolerate bounded blocking (e.g. VTASK_LIVE_BLOCK work
|
||||
* such as promotion) -- never call from iomgr or delaymgr. Same CAS-claim
|
||||
* coalescing as persist_apply_pump_kick: a losing concurrent caller is a
|
||||
* safe no-op. See persist.c for the promotion-race rationale. */
|
||||
void persist_apply_pump_run_sync(persist_t* p);
|
||||
|
||||
int persist_job_hook_fun(int type, int situation, int detail,
|
||||
void* cls, va_list ap);
|
||||
int persist_queue_hook_fun(int type, int situation, int detail,
|
||||
|
|
|
|||
|
|
@ -1172,6 +1172,18 @@ int iomgr_init(iomgr_t* iomgr)
|
|||
* any socket I/O, so this protects controller, worker, and scheduler. */
|
||||
signal(SIGPIPE, SIG_IGN);
|
||||
|
||||
/* bug-1406: establish every iomgr_t invariant here rather than assuming
|
||||
* the caller already zeroed the struct (do_endpoint_init follows this
|
||||
* same fully-explicit pattern for endpoint_t, a few lines above in this
|
||||
* file). Every production caller embeds iomgr_t inside a calloc'd/static
|
||||
* context struct, which previously masked this gap, but a caller that
|
||||
* stack-allocates an iomgr_t without a memset (test_ha_vep_teardown.c,
|
||||
* 4 call sites) left ssl_ctx as garbage; iomgr_fini's `!= NULL` guard then
|
||||
* passed on that garbage pointer and SSL_CTX_free() segfaulted on a wild
|
||||
* address. */
|
||||
iomgr->ssl_ctx = NULL;
|
||||
iomgr->worker = 0;
|
||||
|
||||
mylock_init(&(iomgr->lock));
|
||||
iomgr->epoll_fd = epoll_create1(0);
|
||||
if (iomgr->epoll_fd < 0) {
|
||||
|
|
|
|||
|
|
@ -383,26 +383,44 @@ assert_no_loss() {
|
|||
# Assert the cluster currently lists EXACTLY N jobs (Req 5.1/5.2), gated on the
|
||||
# xce_test exit code. Used by the cold-restart-all recovery phase. Returns 0 iff
|
||||
# the live count equals N.
|
||||
#
|
||||
# Retries up to 3 times (2s settle sleep between) before declaring a mismatch:
|
||||
# bug-1396's original "extra job" observation and a fresh repro of this exact
|
||||
# check both trace to the SAME client-side race as bug-1422 (see
|
||||
# .wolf/buglog.json) — right after a fresh election (e.g. this phase's cold
|
||||
# restart), libxce's failover retry loop can exhaust its budget without ever
|
||||
# reaching the new primary and surface a stale E_XCE_STANDBY from whichever
|
||||
# standby it last hit, which reads here as "xce_joblist: Unknown error 10014"
|
||||
# / "could not determine job count" rather than a real count. Confirmed via a
|
||||
# live repro: the aggregate expect-jobs call failed this way while the very
|
||||
# next per-id assert_no_loss check (16 individual expect-job calls) found
|
||||
# every job present. A count still wrong after a real settle + 2 retries is a
|
||||
# genuine mismatch, not this race.
|
||||
assert_job_count() {
|
||||
local label="$1" expected="$2" rc
|
||||
xce_test_script "$(printf 'set masters %s\nclient\nexpect-jobs %s\n' \
|
||||
"$MASTERS" "$expected")"
|
||||
rc=$?
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
pass "$label: cluster lists exactly $expected jobs"
|
||||
return 0
|
||||
else
|
||||
# bug-1396: a bare count mismatch isn't enough to diagnose an extra-job
|
||||
# anomaly — dump the live joblist and the harness's own ACKED_FILE
|
||||
# ledger side by side so the diverging job ID is visible in the log
|
||||
# instead of only the count.
|
||||
yellow " $label: job count mismatch — dumping live joblist vs ACKED_FILE for diagnosis"
|
||||
xce_test_script "$(printf 'set masters %s\nclient\njoblist\n' "$MASTERS")"
|
||||
echo " --- ACKED_FILE (tracked committed jobs, deduplicated) ---"
|
||||
sort -u "$ACKED_FILE" | sed 's/^/ /'
|
||||
fail "$label: job count != $expected after recovery (see joblist/ACKED_FILE dump above)"
|
||||
return 1
|
||||
fi
|
||||
local label="$1" expected="$2" rc attempt
|
||||
for attempt in 1 2 3; do
|
||||
xce_test_script "$(printf 'set masters %s\nclient\nexpect-jobs %s\n' \
|
||||
"$MASTERS" "$expected")"
|
||||
rc=$?
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
pass "$label: cluster lists exactly $expected jobs"
|
||||
return 0
|
||||
fi
|
||||
if [ "$attempt" -lt 3 ]; then
|
||||
yellow " $label attempt $attempt/3: count check failed (likely post-election client retry-storm race, bug-1422 class) — settling 2s and retrying"
|
||||
sleep 2
|
||||
fi
|
||||
done
|
||||
# bug-1396: a bare count mismatch isn't enough to diagnose an extra-job
|
||||
# anomaly — dump the live joblist and the harness's own ACKED_FILE
|
||||
# ledger side by side so the diverging job ID is visible in the log
|
||||
# instead of only the count.
|
||||
yellow " $label: job count mismatch after 3 attempts — dumping live joblist vs ACKED_FILE for diagnosis"
|
||||
xce_test_script "$(printf 'set masters %s\nclient\njoblist\n' "$MASTERS")"
|
||||
echo " --- ACKED_FILE (tracked committed jobs, deduplicated) ---"
|
||||
sort -u "$ACKED_FILE" | sed 's/^/ /'
|
||||
fail "$label: job count != $expected after recovery (see joblist/ACKED_FILE dump above)"
|
||||
return 1
|
||||
}
|
||||
|
||||
# ── soak mode (original racy reproduction) ────────────────────────────────────
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue