forked from lei.chen/myxce
feat(job-array-single-wal-record): make apply pump respect apply failures
persist_apply_pump_run() no longer discards apply_committed_entry()'s return value. On a negative rc for any WAL type (not just the new array type), the walk halts at that LSN — no last_applied_lsn advance, no pending-apply hand-off — logs a loud error, and retries from the same position on the next kick, mirroring the existing apply_pump_read_entry failure branch's shape. Investigated whether this newly turns the pre-existing job-fact -ENOENT path into a permanent-wedge hazard: it does not, since c->jobmgr never deletes entries and facts are only ever applied after their causally-prior submit already succeeded, so a live -ENOENT there can only mean genuine divergence — halting is the correct behavior. Task 4.1 of job-array-single-wal-record (Requirement 9.4).
This commit is contained in:
parent
107e3839d9
commit
cc7e99c2f7
3 changed files with 236 additions and 2 deletions
|
|
@ -82,7 +82,7 @@
|
|||
- _Boundary: Array Apply Expansion_
|
||||
|
||||
- [ ] 4. Live Apply Resource Fence
|
||||
- [ ] 4.1 Make the apply pump respect apply failures
|
||||
- [x] 4.1 Make the apply pump respect apply failures
|
||||
- Stop discarding the array-expansion (and every other WAL type's) apply outcome; on a failure, halt the apply walk at that position without advancing past it, and log a loud, operator-visible error naming the stuck position
|
||||
- Observable: a simulated apply failure at a given position causes the walk to stop exactly there and not proceed to later entries
|
||||
- _Requirements: 9.4_
|
||||
|
|
@ -175,3 +175,4 @@
|
|||
- **Task 1.3**: `makeJobsub()`/`makeJobstartinfo()` now return `-ENODEV` (not a generic `-EINVAL`) when the interning table isn't initialized — if a later task's error handling branches on `job_init()`'s return value, `-ENODEV` is a distinct, checkable signal for "table not ready" vs. other construction failures.
|
||||
- **Task 3.3**: `apply_wal_entry()` (persist.c:2574) is dead code in production — confirmed via exhaustive grep (zero call sites outside its own declaration/definition/comments) and via `ha.c:8024`'s `ha_standby_advance_committed_and_apply` (dated 2026-07-10), whose own doc comment explicitly says standby entity materialization now goes through the SAME `apply_committed_entry()`/`persist_apply_pump_kick` pump the primary uses, not `apply_wal_entry()`. The persist.c:3722 comment claiming otherwise predates that refactor and was never updated (design.md corrected accordingly). Requirement 3.1/3.2/4.1 are fully delivered by task 3.2 alone. Task 3.3's code was kept (not discarded) as defensive parity with `WAL_JOB_SUBMIT`'s existing, also-dead case in the same switch — it is tested and correct, just not currently reachable in production. If `apply_wal_entry()` is ever revived or removed entirely in a future spec, this task's case should move or be deleted with it.
|
||||
- **Task 3.1**: `restore_job_array()`'s atomicity has one narrow, documented exception (design.md updated to match): the final bulk-insert-into-`c->jobmgr` pass can itself fail on allocation/growth exhaustion (`c->jobmgr` grows via `realloc()`; no capacity-reservation primitive exists anywhere in `kv.c`/`kvmgr.c`/`zarray.c` to pre-check this before mutating). On this specific failure, the function never silently claims success — it returns a negative errno — but an already-inserted prefix `[0, inserted)` can be left genuinely, permanently visible in `c->jobmgr` (no `key_del` exists to undo it), and that prefix is NEVER included in `out_new_jobs`, so `HOOK_DETAIL_JOB_NEW` never fires for it, not on this attempt or any later successful retry of the same LSN. **Tasks 3.2/4.x must account for this**: those elements are individually addressable (status/cancel/completion work normally) but permanently un-pushed to the scheduler/API server unless something later re-derives and re-notifies them — this is a real gap in the current design, not yet solved, and task 4.x's Live Apply Resource Fence design should explicitly decide whether/how to handle it (e.g. detecting this residual state and re-notifying on self-heal) rather than only handling the construction-phase failure case.
|
||||
- **Task 4.1**: `persist_apply_pump_run()` now halts the walk (no `last_applied_lsn` advance) on ANY negative return from `apply_committed_entry()`, not just the new array type — this is a general fix. Investigated whether this newly exposes a hazard for the pre-existing job-state fact entries (`WAL_JOB_START/FINISH/FAIL/UPDATE`), which have their own real `-ENOENT` branch when the target jobid isn't in `c->jobmgr`: confirmed NOT a hazard, because `c->jobmgr` (`xkvmgr_type`) has no `key_del` (jobs are never removed for the process lifetime, corroborated by `job.c:428,455,645`), and this task's own strict sequential-no-skip apply means a fact's LSN is only reached after its causally-prior submit already applied. So a live `-ENOENT` on a fact can now only mean genuine corruption/divergence — halting is correct, not a new footgun. Independent reviewer confirmed this via `test_term_safety_adapter.c:2039-2043`'s synthetic (non-production-representative) trigger. design.md's "Existing Architecture Analysis" table row overclaims ("every existing WAL type... returns 0 or positive in practice") — technically inaccurate re: the fact-ENOENT branch — left as-is since the persist.c code comment at the site already documents the exception correctly; low-severity, non-blocking.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue