docs(kiro): reconcile spec metadata, promote 3 new specs, fix template path
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
3f58a3636b
commit
ce0ee78b0f
SSH key fingerprint:
SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
18 changed files with 467 additions and 114 deletions
|
|
@ -27,9 +27,9 @@ Otherwise, load all necessary context:
|
|||
- `.kiro/specs/{feature}/research.md` (if exists, contains gap analysis from `/kiro-validate-gap`)
|
||||
- Core steering context: `product.md`, `tech.md`, `structure.md`
|
||||
- Additional steering files only when directly relevant to requirement coverage, architecture boundaries, integrations, runtime prerequisites, security/performance constraints, or team conventions that affect implementation readiness
|
||||
- `templates/specs/design.md` from this skill's directory for document structure
|
||||
- `.kiro/settings/templates/specs/design.md` for document structure
|
||||
- Read `rules/design-principles.md` from this skill's directory for design principles
|
||||
- `templates/specs/research.md` from this skill's directory for discovery log structure
|
||||
- `.kiro/settings/templates/specs/research.md` for discovery log structure
|
||||
|
||||
**Validate requirements approval**:
|
||||
- If auto-approve flag is true: Auto-approve requirements in spec.json
|
||||
|
|
@ -157,7 +157,7 @@ Provide brief summary in the language specified in spec.json:
|
|||
|
||||
**Format**: Concise Markdown (under 200 words) - this is the command output, NOT the design document itself
|
||||
|
||||
**Note**: The actual design document follows `templates/specs/design.md` (this skill's directory) structure.
|
||||
**Note**: The actual design document follows `.kiro/settings/templates/specs/design.md` structure.
|
||||
|
||||
## Safety & Fallback
|
||||
|
||||
|
|
@ -174,7 +174,7 @@ Provide brief summary in the language specified in spec.json:
|
|||
- **Suggested Action**: "Run `/kiro-spec-requirements {feature}` to generate requirements first"
|
||||
|
||||
**Template Missing**:
|
||||
- **User Message**: "Template file missing at `templates/specs/design.md` in this skill's directory"
|
||||
- **User Message**: "Template file missing at `.kiro/settings/templates/specs/design.md`"
|
||||
- **Suggested Action**: "Check repository setup or restore template file"
|
||||
- **Fallback**: Use inline basic structure with warning
|
||||
|
||||
|
|
|
|||
|
|
@ -17,8 +17,8 @@ Generate a unique feature name from the project description ($ARGUMENTS) and ini
|
|||
3. **Check Uniqueness**: Verify `.kiro/specs/` for naming conflicts. If the directory already exists with only `brief.md` (no `spec.json`), use that directory (discovery created it).
|
||||
4. **Create Directory**: `.kiro/specs/[feature-name]/` (skip if already exists from discovery)
|
||||
5. **Initialize Files Using Templates**:
|
||||
- Read `templates/specs/init.json` from this skill's directory
|
||||
- Read `templates/specs/requirements-init.md` from this skill's directory
|
||||
- Read `.kiro/settings/templates/specs/init.json`
|
||||
- Read `.kiro/settings/templates/specs/requirements-init.md`
|
||||
- Replace placeholders:
|
||||
- `{{FEATURE_NAME}}` → generated feature name
|
||||
- `{{TIMESTAMP}}` → current ISO 8601 timestamp
|
||||
|
|
@ -46,6 +46,6 @@ Provide output in the language specified in `spec.json` with the following struc
|
|||
|
||||
## Safety & Fallback
|
||||
- **Ambiguous Feature Name**: If feature name generation is unclear, propose 2-3 options and ask user to select
|
||||
- **Template Missing**: If template files don't exist in this skill's `templates/specs/`, report error with specific missing file path and suggest checking skill installation
|
||||
- **Template Missing**: If template files don't exist in `.kiro/settings/templates/specs/`, report error with specific missing file path and suggest checking repository setup
|
||||
- **Directory Conflict**: If feature name already exists, append numeric suffix (e.g., `feature-name-2`) and notify user of automatic conflict resolution
|
||||
- **Write Failure**: Report error with specific path and suggest checking permissions or disk space
|
||||
|
|
|
|||
|
|
@ -72,10 +72,10 @@ Execute these 4 phases in order:
|
|||
|
||||
5. **Initialize Files from Templates**:
|
||||
|
||||
a. Read templates from this skill's directory:
|
||||
a. Read templates:
|
||||
```
|
||||
- templates/specs/init.json
|
||||
- templates/specs/requirements-init.md
|
||||
- .kiro/settings/templates/specs/init.json
|
||||
- .kiro/settings/templates/specs/requirements-init.md
|
||||
```
|
||||
|
||||
b. Replace placeholders:
|
||||
|
|
@ -230,7 +230,7 @@ Next Steps:
|
|||
### Error Scenarios
|
||||
|
||||
**Template Missing**:
|
||||
- Check `templates/specs/` in this skill's directory exists
|
||||
- Check `.kiro/settings/templates/specs/` exists
|
||||
- Report specific missing file
|
||||
- Exit with error
|
||||
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ Otherwise, load all necessary context:
|
|||
### Step 2: Read Guidelines
|
||||
- Read `rules/ears-format.md` from this skill's directory for EARS syntax rules
|
||||
- Read `rules/requirements-review-gate.md` from this skill's directory for pre-write review criteria
|
||||
- Read `templates/specs/requirements.md` from this skill's directory for document structure
|
||||
- Read `.kiro/settings/templates/specs/requirements.md` for document structure
|
||||
|
||||
#### Parallel Research (subagent dispatch)
|
||||
|
||||
|
|
|
|||
|
|
@ -39,7 +39,7 @@ Otherwise, load all necessary context:
|
|||
|
||||
- Read `rules/tasks-generation.md` from this skill's directory for principles
|
||||
- Read `rules/tasks-parallel-analysis.md` from this skill's directory for parallel judgement criteria
|
||||
- Read `templates/specs/tasks.md` from this skill's directory for format (supports `(P)` markers)
|
||||
- Read `.kiro/settings/templates/specs/tasks.md` for format (supports `(P)` markers)
|
||||
|
||||
#### Parallel Research
|
||||
|
||||
|
|
@ -176,9 +176,9 @@ Provide brief summary in the language specified in spec.json:
|
|||
- **Suggested Action**: "Refine requirements.md or design.md, then re-run `/kiro-spec-tasks {feature}`"
|
||||
|
||||
**Template/Rules Missing**:
|
||||
- **User Message**: "Template or rules files missing in this skill's `templates/specs/` or `rules/` directory"
|
||||
- **User Message**: "Template or rules files missing in `.kiro/settings/`"
|
||||
- **Fallback**: Use inline basic structure with warning
|
||||
- **Suggested Action**: "Check skill installation or restore template files"
|
||||
- **Suggested Action**: "Check repository setup or restore template files"
|
||||
- **Missing Numeric Requirement IDs**:
|
||||
- **Stop Execution**: All requirements in requirements.md MUST have numeric IDs. If any requirement lacks a numeric ID, stop and request that requirements.md be fixed before generating tasks.
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ You are a specialized skill for creating custom steering documents beyond core f
|
|||
|
||||
If steering context is already available from conversation, skip redundant file reads.
|
||||
Otherwise:
|
||||
- Check `templates/steering-custom/` in this skill's directory for available templates
|
||||
- Check `.kiro/settings/templates/steering-custom/` for available templates
|
||||
- Read `rules/steering-principles.md` from this skill's directory for steering principles
|
||||
|
||||
## Workflow
|
||||
|
|
@ -37,7 +37,7 @@ Otherwise:
|
|||
- Specific requirements or patterns to document
|
||||
|
||||
2. **Check if template exists**:
|
||||
- Load from `templates/steering-custom/{name}.md` in this skill's directory if available
|
||||
- Load from `.kiro/settings/templates/steering-custom/{name}.md` if available
|
||||
- Use as starting point, customize based on project
|
||||
|
||||
3. **Analyze codebase** (JIT) for relevant patterns:
|
||||
|
|
@ -60,7 +60,7 @@ After all parallel research completes, synthesize findings for steering document
|
|||
|
||||
## Available Templates
|
||||
|
||||
Templates available in `templates/steering-custom/` (this skill's directory):
|
||||
Templates available in `.kiro/settings/templates/steering-custom/`:
|
||||
|
||||
1. **api-standards.md** - REST/GraphQL conventions, error handling
|
||||
2. **testing.md** - Test organization, mocking, coverage
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ You are a specialized skill for maintaining `.kiro/steering/` as persistent proj
|
|||
|
||||
If steering context is already available from conversation, skip redundant file reads.
|
||||
|
||||
- For Bootstrap mode: Read templates from `templates/steering/` in this skill's directory
|
||||
- For Bootstrap mode: Read templates from `.kiro/settings/templates/steering/`
|
||||
- For Sync mode: Read all existing `.kiro/steering/*.md` files
|
||||
- Read `rules/steering-principles.md` from this skill's directory for steering principles
|
||||
|
||||
|
|
@ -45,7 +45,7 @@ Check `.kiro/steering/` status:
|
|||
|
||||
## Bootstrap Flow
|
||||
|
||||
1. Load templates from `templates/steering/` in this skill's directory
|
||||
1. Load templates from `.kiro/settings/templates/steering/`
|
||||
2. Analyze codebase (JIT):
|
||||
|
||||
#### Parallel Research
|
||||
|
|
|
|||
|
|
@ -15,14 +15,15 @@ Scheduler, worker, and apiserver each connect to the controller cluster today us
|
|||
- apiserver returns a clear, fast error to its own API clients while its backend connection is down, instead of hanging (Req 4).
|
||||
- Every connect/reconnect attempt and outcome is logged per-candidate (Req 5).
|
||||
- apiserver accepts the identical candidate-list syntax scheduler/worker already accept (Req 6).
|
||||
- Scheduler decision publishing (`xcecli_jobdispatch`/`xcecli_set_pend_reason`) never silently drops a decision on retry-budget exhaustion — it keeps retrying that decision until success or shutdown, so scheduler-local and controller-observed job state cannot permanently diverge (Req 7).
|
||||
|
||||
### Non-Goals
|
||||
- Changing `xce_test` (it is already the reference-correct client).
|
||||
- Changing `xce_test` (it is already the reference-correct client) — its clamp already references `N_MAX_MASTERS` symbolically (`test.c:372-378`), so it auto-absorbs the constant reconciliation below with zero code change.
|
||||
- Completing or adopting the separate, unused `xce_connect_cluster()`/`xce_reconnect_*` primitive in `cluster.c` — superseded by the mechanism this design adopts; left untouched.
|
||||
- Raising the `N_MAX_MASTERS` (3) cap or reconciling it with `MAX_HA_CANDIDATES` (4) — pre-existing constraint, replicated (clamp + WARN) exactly as `xce_test` already does.
|
||||
- Wiring CA-cert/TLS trust into the real connect path — `xce_client_config_t` carries no such field today and this feature does not add one; TLS is established the same way it already is.
|
||||
- Controller-side changes (redirect-reply population, peer authentication) — already shipped by `libxce-failover`.
|
||||
- Recovering worker job-state lost during a disconnect window — owned by `worker-rejoin-resync`.
|
||||
- Persisting queued-but-unpublished scheduler decisions across a scheduler process crash/restart (Req 7 AC6) — Req 7's fix keeps a running scheduler process from giving up early on a network retry; it adds no WAL/persistence for the in-memory `decisions`/`jbuf` queues, so a scheduler crash mid-outage still loses whatever was queued, exactly as it does today.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
|
|
@ -32,14 +33,16 @@ Scheduler, worker, and apiserver each connect to the controller cluster today us
|
|||
- apiserver's candidate-list config parsing (routed through the existing C parser via CGO instead of a second Go implementation).
|
||||
- apiserver's client-facing fail-fast behavior while its backend connection is degraded.
|
||||
- Connect/reconnect diagnostic logging in all three components.
|
||||
- Scheduler's decision-publish retry-until-success-or-shutdown wrapper around `xcecli_jobdispatch`/`xcecli_set_pend_reason` (Req 7) — closes the silent-drop gap found in the 2026-08-10 design validation.
|
||||
- **Reconciling `N_MAX_MASTERS` (`client.h`) to derive from `MAX_HA_CANDIDATES` (`config.h`)** (Req 1 AC7, scope expanded 2026-08-10 at explicit user direction — supersedes this design's earlier Non-Goal treating the mismatch as pre-existing/out-of-scope): `config.h` becomes the single source of truth for "max configured controller candidates." Confirmed safe by source audit — no wire-protocol array assumes 3 (`masters[]` is never serialized; the controller's own peer list is a dynamic capnp `List(CtrlPeer)`), and `config.h` has no include-cycle risk (it depends only on `<stdint.h>`; `client.h` already transitively sees `MAX_HA_CANDIDATES` via `xceapi.h`/`xce/types.h` → `config.h`).
|
||||
- **Eliminating the legacy single-host code path in worker, scheduler, and apiserver in favor of a single unified pipeline** (Req 1 AC5, redefined 2026-08-10): today all three synthesize a `nmasters=1` `xce_client_config_t` directly from legacy fields, bypassing `xce_cluster_config_t`/`xce_parse_peers`/`xce_cluster_config_validate`/ConfigConverter entirely (`controller_connector.c:129-160`'s `init_client_config()`, `scheduler.c:613`'s single-host `else` branch, `clib.go:79-101`'s `tryConnect()` direct-CGO-struct path). This spec now closes that parallel path: a single controller is the `candidates_count=1` case of the exact same validated pipeline, not a separately-maintained mechanism.
|
||||
|
||||
### Out of Boundary
|
||||
- `xce_test` and its `masters` setting (unchanged, reference implementation).
|
||||
- `libxce`'s failover mechanism itself (`client.c`, `client_internal.h`, `client_failover.c`) — consumed as-is, not modified.
|
||||
- `libxce`'s failover **mechanism** (`client.c`'s connect/rotate/retry/backoff logic, `client_internal.h`, `client_failover.c`) — consumed as-is, not modified. Distinct from `client.h`'s `N_MAX_MASTERS` **constant**, which this spec does change (above) — the mechanism and the struct-layout constant are different surfaces with different boundaries.
|
||||
- `xce_connect_cluster()`/`xce_reconnect_*` (unused, untouched).
|
||||
- Controller-side membership/redirect logic.
|
||||
- Worker job-state resync content on rejoin (`worker-rejoin-resync`).
|
||||
- `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` value changes.
|
||||
- TLS/CA-cert plumbing into the real connect path.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
|
@ -48,7 +51,7 @@ Scheduler, worker, and apiserver each connect to the controller cluster today us
|
|||
- Existing per-component candidate-list CLI/YAML surfaces (`--peers`, `--ca-cert`, `backend.peers`) — unmodified externally; only how they're consumed changes.
|
||||
|
||||
### Revalidation Triggers
|
||||
- Any change to `xce_client_config_t`'s layout or `N_MAX_MASTERS`.
|
||||
- Any change to `xce_client_config_t`'s layout, or any future change to `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` that reintroduces divergence between them (this spec establishes them as one value — a regression that redefines `N_MAX_MASTERS` independently of `config.h` again should re-trigger review).
|
||||
- Any change to `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL` retry/classification semantics (`libxce-failover`).
|
||||
- Any change to `xce_parse_peers`/`xce_cluster_config_validate` syntax or validation rules (apiserver's CGO-based parser would need re-verification).
|
||||
- `worker-rejoin-resync` landing — worker's rejoin callback becomes the natural place to also trigger job-state resync; this spec's callback is a no-op-beyond-rejoin placeholder for that.
|
||||
|
|
@ -115,7 +118,7 @@ Verified directly against `client_failover.c:341-411`: `XCECLI_FAILOVER_CALL` ho
|
|||
|
||||
This directly constrains every component's rejoin implementation in this spec:
|
||||
- The callback **registered with the library** (worker's `checking_connection`→`xcecli_hello` path, scheduler's `xcecli_jobdispatch` path, apiserver's write-call path) must do **no I/O and call no `xcecli_*` function** — it may only set a lightweight "needs rejoin" flag (a plain bool/atomic on the consumer side) and return `0` so the interrupted operation resumes normally.
|
||||
- Each component's own calling code, **after** the triggering `xcecli_*` call returns (at which point the library has released the lock), checks and clears that flag and issues `xcecli_join(...)` **as a separate, subsequent call** — safe, since the lock is free by then.
|
||||
- Each component's own calling code, **after** the triggering `xcecli_*` call returns (at which point the library has released the lock), checks that flag and, if set, issues `xcecli_join(...)` **as a separate, subsequent call** — safe, since the lock is free by then. **The flag is cleared only once that `xcecli_join` call itself succeeds, never merely because it was attempted** (Req 2 AC5, gap found in the 2026-08-10 design validation's second pass): `xcecli_join` is itself `XCECLI_FAILOVER_CALL`-wrapped and can exhaust its own retry budget and return `-E_XCE_NOT_READY` just like any other wrapped call. Clearing the flag unconditionally before or regardless of that outcome would leave the component transport-reconnected but not cluster-joined, with nothing left to retry the join until the next full disconnect/reconnect cycle happens to fire the rejoin callback again — an indefinite, silent orphaning, not a bounded delay. On failure the flag stays set, so the component's next naturally-occurring opportunity (worker: next `xcecli_hello` return; scheduler: next decision-publish return, itself now retried indefinitely per Req 7; apiserver: next backend-call wrapper return) re-attempts the join.
|
||||
- apiserver's heartbeat-triggered path (see below) is unaffected by this constraint — it already detects recovery and calls the rejoin logic **after** `xcecli_heartbeat` has returned (`XCECLI_FF_CALL` also releases its lock before returning), so it was never at risk.
|
||||
|
||||
### Technology Stack
|
||||
|
|
@ -130,46 +133,89 @@ This directly constrains every component's rejoin implementation in this spec:
|
|||
|
||||
### Modified / New Files
|
||||
```
|
||||
src/include/client.h # fix: N_MAX_MASTERS derives from
|
||||
# config.h's MAX_HA_CANDIDATES (3->4)
|
||||
# instead of an independent literal (Req 1.7)
|
||||
src/include/xce/cluster.h # + declare xce_cluster_config_to_client_config()
|
||||
src/lib/cluster.c # + implement xce_cluster_config_to_client_config()
|
||||
src/lib/testsuite/test_cluster.c # + unit tests for the new helper
|
||||
src/lib/testsuite/test_libxce_retry.c # fix: fo_make_client_masters() currently
|
||||
# hardcodes 3 literal host/port entries then
|
||||
# copies N_MAX_MASTERS slots -- update to match
|
||||
# the raised N_MAX_MASTERS, not silently copy
|
||||
# an uninitialized 4th slot
|
||||
|
||||
src/worker/src/controller_connector.c # rewrite: delete per-candidate loop in
|
||||
# xcecd_join_cluster/init_client_config;
|
||||
# single init+start+join using full
|
||||
# candidate list; register rejoin callback
|
||||
src/worker/src/controller_connector.c # rewrite: delete init_client_config()'s
|
||||
# standalone xce_client_config_t construction
|
||||
# and xcecd_join_cluster()'s separate
|
||||
# single-host-fallback branch (both bypassed
|
||||
# xce_cluster_config_t entirely) -- single
|
||||
# init+start+join via the one synthesize-
|
||||
# validate-convert-connect sequence, whether
|
||||
# the candidate list came from --peers or was
|
||||
# synthesized as one candidate (Req 1.5);
|
||||
# register rejoin callback
|
||||
src/worker/src/main.c # fix: populate cc_ctx->cluster_cfg BEFORE
|
||||
# xcecd_start() (removes init-order bug)
|
||||
# xcecd_start() (removes init-order bug);
|
||||
# fix hardcoded `if (candidates_count > 3)`
|
||||
# (main.c:414) to reference N_MAX_MASTERS
|
||||
# symbolically instead of a literal 3, so it
|
||||
# tracks the raised value automatically
|
||||
src/worker/include/controller_connector.h # + rejoin callback function declaration
|
||||
|
||||
src/scheduler/src/scheduler.c # rewrite: sched_init uses the new helper +
|
||||
# single init+start+join; delete inline
|
||||
# per-candidate loop; register rejoin callback
|
||||
src/scheduler/src/main.c # no ordering change needed (already correct);
|
||||
# simplified call site only
|
||||
# per-candidate loop AND the separate
|
||||
# "single-host path (original)" else branch
|
||||
# (scheduler.c:613, bypassed
|
||||
# xce_cluster_config_t entirely) -- single-host
|
||||
# is now candidates_count=1 through the same
|
||||
# sequence (Req 1.5); register rejoin callback;
|
||||
# sched_publish_dispatch/sched_publish_pend_reason
|
||||
# gain a retry-until-success-or-shutdown wrapper
|
||||
# around their xcecli_* call (Req 7)
|
||||
src/scheduler/src/main.c # fix hardcoded `if (candidates_count > 3)`
|
||||
# (main.c:314) to reference N_MAX_MASTERS
|
||||
# symbolically, same as worker's fix above;
|
||||
# no ordering change needed (already correct)
|
||||
|
||||
src/frontend/apiserver/backend/clib.go # rewrite RegisterToBackend: single connect
|
||||
# using the full candidate list via CGO calls
|
||||
# into xce_parse_peers/xce_cluster_config_validate/
|
||||
# xce_cluster_config_to_client_config; cgo
|
||||
# preamble gains a static C shim (mirrors the
|
||||
# existing xce_backend_handler/go_push_handler
|
||||
# pattern) that calls the reconnect.go export,
|
||||
# passed to xcecli_set_reconnect_handler; the
|
||||
# heartbeat tick callback explicitly re-invokes
|
||||
# the same rejoin path on recovery (see below)
|
||||
src/frontend/apiserver/backend/clib.go # rewrite RegisterToBackend: builds exactly ONE
|
||||
# xce_cluster_config_t regardless of source
|
||||
# (config.Peers parsed via xce_parse_peers, or
|
||||
# a synthesized single candidate from
|
||||
# config.Address/config.Port when Peers is
|
||||
# empty -- Req 1.5) then the same
|
||||
# validate/xce_cluster_config_to_client_config/
|
||||
# connect sequence either way; deletes
|
||||
# tryConnect()'s direct C.xce_client_config_t
|
||||
# construction and the Go-level per-peer loop
|
||||
# entirely; cgo preamble gains a static C shim
|
||||
# (mirrors the existing
|
||||
# xce_backend_handler/go_push_handler pattern)
|
||||
# that calls the reconnect.go export, passed to
|
||||
# xcecli_set_reconnect_handler
|
||||
src/frontend/apiserver/backend/peers.go # DELETED: hand-written parsePeer() retired in
|
||||
# favor of the CGO-routed xce_parse_peers (Req 6.1)
|
||||
src/frontend/apiserver/backend/health.go # NEW: backend-degraded circuit-breaker state
|
||||
src/frontend/apiserver/backend/reconnect.go # NEW: exported Go rejoin logic (re-sends
|
||||
# xcecli_join for HostRole_apiserver, clears
|
||||
# health.go state); called from clib.go's C
|
||||
# shim AND directly from the heartbeat callback
|
||||
src/frontend/apiserver/backend/heartbeat.go # MODIFIED: tick callback detects a
|
||||
# degraded-to-healthy transition after
|
||||
# xcecli_heartbeat succeeds and explicitly
|
||||
# triggers reconnect.go's rejoin logic
|
||||
src/frontend/apiserver/utils/config.go # no schema change (Peers already exists)
|
||||
src/frontend/apiserver/backend/heartbeat.go # MODIFIED: tick callback sets a needs_rejoin
|
||||
# flag on a failure->success transport
|
||||
# transition (FF_CALL never sets it itself),
|
||||
# then on every subsequent tick where
|
||||
# xcecli_heartbeat succeeds AND needs_rejoin is
|
||||
# still true (not gated on
|
||||
# ApiserverBackendHealth.IsDegraded(), which has
|
||||
# an independent TTL -- Req 2.5) retries
|
||||
# reconnect.go's rejoin function until it
|
||||
# actually succeeds
|
||||
src/frontend/apiserver/utils/config.go # no schema change (Peers/Address/Port already exist)
|
||||
|
||||
src/worker/testsuite/test_e2e_multimaster_join.c # NEW: candidate-list join + reconnect E2E
|
||||
src/worker/testsuite/test_e2e_multimaster_join.c # NEW: candidate-list join + reconnect E2E;
|
||||
# also covers the unified single-host path
|
||||
# (no --peers) hitting the same code (Req 1.5)
|
||||
src/scheduler/testsuite/test_e2e_multimaster_join.c # NEW: same, scheduler
|
||||
src/frontend/apiserver/backend/health_test.go # NEW: circuit-breaker unit tests
|
||||
src/frontend/apiserver/backend/clib_test.go # + candidate-list parity tests (existing file)
|
||||
|
|
@ -231,7 +277,7 @@ sequenceDiagram
|
|||
Ctx-->>Daemon: rejoin complete
|
||||
```
|
||||
|
||||
Key decision reflected in the diagram: the rejoin callback invoked **by the library** never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual `xcecli_join` call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: allocator thread; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way.
|
||||
Key decision reflected in the diagram: the rejoin callback invoked **by the library** never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual `xcecli_join` call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: the main driver-loop thread, inside `sched_publish_dispatch` immediately after `xcecli_jobdispatch` returns — **not** the allocator thread, see SchedulerControllerClient below; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way.
|
||||
|
||||
### apiserver fail-fast while degraded (Req 4)
|
||||
|
||||
|
|
@ -275,12 +321,19 @@ sequenceDiagram
|
|||
HB->>Ctx: xcecli_heartbeat (fire-and-forget)
|
||||
Ctx->>Ctx: connection known-dead: best-effort rotate + reconnect
|
||||
Ctx->>NewPrimary: reconnected transport only
|
||||
Ctx-->>HB: rc success (was failing, now succeeds)
|
||||
HB->>HB: detect failure-to-success transition
|
||||
HB->>Rejoin: explicitly trigger rejoin
|
||||
Ctx-->>HB: rc success (transport reconnected)
|
||||
Note over HB: was failing, now succeeds -> set needs_rejoin=true (FF_CALL never sets it for us)
|
||||
HB->>HB: needs_rejoin still true?
|
||||
HB->>Rejoin: trigger rejoin
|
||||
Rejoin->>NewPrimary: xcecli_join(HostRole_apiserver)
|
||||
NewPrimary-->>Rejoin: joined
|
||||
Rejoin->>Health: MarkHealthy()
|
||||
alt join succeeds
|
||||
NewPrimary-->>Rejoin: joined
|
||||
Rejoin->>HB: clear needs_rejoin
|
||||
Rejoin->>Health: MarkHealthy()
|
||||
else join's own retry budget exhausted
|
||||
NewPrimary--xRejoin: -E_XCE_NOT_READY
|
||||
Note over HB: needs_rejoin stays true (independent of Health's TTL-governed state) — next tick retries
|
||||
end
|
||||
```
|
||||
|
||||
This path exists because `XCECLI_FF_CALL` (fire-and-forget, used by the heartbeat) never invokes the library's own rejoin hook — only the full-retry path (`XCECLI_FAILOVER_CALL`, used by write calls) does. Without this explicit trigger, an idle apiserver instance would repair its TCP connection on every heartbeat tick but never re-establish cluster membership, so push notifications would stay dead until an unrelated write request happened to arrive.
|
||||
|
|
@ -291,23 +344,26 @@ This path exists because `XCECLI_FF_CALL` (fire-and-forget, used by the heartbea
|
|||
|-------------|---------|------------|------------|-------|
|
||||
| 1.1-1.3 | Try each candidate in order at startup | libxce (unchanged), ConfigConverter | `xcecli_start`, `xcecli_join` | Startup connect |
|
||||
| 1.4 | Worker uses full list on FIRST attempt | WorkerControllerConnector | `controller_connector.c` init order fix | Startup connect |
|
||||
| 1.5 | Single/no-candidate mode unchanged | All three | ConfigConverter (`nmasters=1` path) | — |
|
||||
| 1.5 | Single controller is `candidates_count=1` of the one mechanism, not a separate fallback (redefined 2026-08-10, was "unchanged single-host mode") | All three | ConfigConverter (`nmasters=1` case, no branch) — worker's `init_client_config()`/single-host branch, scheduler's single-host `else` branch, and apiserver's `tryConnect()` all deleted | Startup connect |
|
||||
| 1.6 | Clear startup failure if all candidates fail | All three | existing `xcecli_start`/`xcecli_join` return codes | Startup connect |
|
||||
| 1.7 | `MAX_HA_CANDIDATES` (`config.h`) is the single source of truth for max candidates; `N_MAX_MASTERS` derives from it | ConfigConverter (clamp becomes defensive, not normally triggered) | `client.h`'s `N_MAX_MASTERS` `#define` | — |
|
||||
| 2.1-2.3 | Automatic reconnect + retry until success or shutdown | libxce (unchanged), all three's rejoin callbacks | `xcecli_set_reconnect_handler` | Runtime failover |
|
||||
| 2.4 | apiserver resumes push notifications after reconnect (incl. idle-period recovery) | ApiserverBackendBridge | `reconnect.go`'s actual-rejoin function re-sends `xcecli_join(HostRole_apiserver)`, triggered via the shared post-call helper's flag check on write-path reconnect AND explicitly from `heartbeat.go` on idle-period recovery | Runtime failover, apiserver idle-period reconnect |
|
||||
| 3.1-3.2 | Retry/backoff doesn't block other liveness work | WorkerControllerConnector (thread isolation preserved), SchedulerControllerClient (allocator/picker thread separation) | existing thread structure | Runtime failover |
|
||||
| 2.5 | Re-join call itself retries on failure, not treated as one-shot | WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge | flag/degraded-state clears only on `xcecli_join` success, not on mere attempt; apiserver's idle-path checks "still degraded" every tick, not just the failure→success edge | Runtime failover, apiserver idle-period reconnect |
|
||||
| 3.1-3.3 | Retry/backoff doesn't block other liveness work | WorkerControllerConnector (thread isolation, full isolation per AC1), SchedulerControllerClient (bounded, occupancy-dependent buffering via `jbuf`/`decisions` cbufs per AC2 — narrower than worker's guarantee by design, no job lost per Req 7, resolved 2026-08-10) | existing thread structure | Runtime failover |
|
||||
| 4.1-4.2 | apiserver fails fast to clients while degraded | ApiserverBackendHealth | `health.go` | apiserver fail-fast flow |
|
||||
| 5.1-5.2 | Per-candidate connect/reconnect logging | All three | log statements at each rotation | Startup connect, Runtime failover |
|
||||
| 6.1-6.2 | Consistent candidate-list syntax | ApiserverConfigParity | CGO call to `xce_parse_peers`/`xce_cluster_config_validate` | Startup connect |
|
||||
| 7.1-7.6 | Scheduler decision publish survives retry-budget exhaustion without loss | SchedulerControllerClient | `sched_publish_dispatch`/`sched_publish_pend_reason` retry-until-success-or-shutdown wrapper | Runtime failover |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|
||||
|-----------|--------------|--------|---------------|-------------------|-----------|
|
||||
| ConfigConverter | libxce (C) | Convert validated candidate list to client config | 1.1-1.6, 6.1 | `xce_cluster_config_validate` (P0) | Service |
|
||||
| WorkerControllerConnector | worker (C) | Single init/connect/join + rejoin for worker | 1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
|
||||
| SchedulerControllerClient | scheduler (C) | Single init/connect/join + rejoin for scheduler | 1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
|
||||
| ApiserverBackendBridge | apiserver (Go/CGO) | Single init/connect/join for apiserver; config parity | 1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2 | ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0) | Service |
|
||||
| ConfigConverter | libxce (C) | Convert validated candidate list to client config | 1.1-1.7, 6.1 | `xce_cluster_config_validate` (P0) | Service |
|
||||
| WorkerControllerConnector | worker (C) | Single init/connect/join + rejoin for worker | 1.1-1.5, 2.1-2.3, 2.5, 3.1, 3.3, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
|
||||
| SchedulerControllerClient | scheduler (C) | Single init/connect/join + rejoin for scheduler; decision-publish never silently drops on retry-budget exhaustion | 1.1-1.3, 1.5-1.7, 2.1-2.3, 2.5, 3.2-3.3, 5.1-5.2, 7.1-7.6 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
|
||||
| ApiserverBackendBridge | apiserver (Go/CGO) | Single init/connect/join for apiserver; config parity | 1.1-1.7, 2.4-2.5, 5.1-5.2, 6.1-6.2 | ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0) | Service |
|
||||
| ApiserverBackendHealth | apiserver (Go) | Fail-fast circuit breaker for backend-down state | 4.1-4.2 | ApiserverBackendBridge (P0, consumes rejoin/error signals) | State |
|
||||
|
||||
### Shared Library
|
||||
|
|
@ -317,12 +373,14 @@ This path exists because `XCECLI_FF_CALL` (fire-and-forget, used by the heartbea
|
|||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Turn a validated `xce_cluster_config_t` into a fully-populated `xce_client_config_t`, clamped to `N_MAX_MASTERS` with a WARN on truncation |
|
||||
| Requirements | 1.1, 1.2, 1.3, 1.5, 1.6, 6.1 |
|
||||
| Requirements | 1.1, 1.2, 1.3, 1.5, 1.6, 1.7, 6.1 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Pure, socket-free transformation (mirrors `xce_test`'s existing inline logic at `test.c:360-372`, generalized into one reusable function).
|
||||
- Does not call `xce_cluster_config_validate()` itself — callers validate first (existing contract, unchanged) and pass a pre-validated config.
|
||||
- Does not own `from`/`key` (identity/auth) — caller sets those on the output struct after conversion, matching how `xce_test`, worker, and scheduler already source them independently (CLI/env-specific).
|
||||
- **Handles `nmasters=1` as the ordinary case, not a special one** (Req 1 AC5): a one-candidate `cfg` produces a one-master `out` through the identical copy loop as any other count — no branch distinguishes "single controller" from "multi controller" anywhere in this function.
|
||||
- **The `N_MAX_MASTERS` clamp is now a defensive safety net, not a normally-triggered path** (Req 1 AC7): since `client.h`'s `N_MAX_MASTERS` now derives from `config.h`'s `MAX_HA_CANDIDATES`, and `xce_cluster_config_validate()` already rejects `candidates_count > MAX_HA_CANDIDATES` before a config ever reaches this function, `cfg->candidates_count > N_MAX_MASTERS` cannot occur via the validated-input contract this function documents as its precondition. The clamp+WARN stays in place as a guard against a caller that skips validation or a future re-divergence of the two constants — it is intentionally not removed, just no longer load-bearing in the normal path.
|
||||
|
||||
**Dependencies**
|
||||
- Inbound: WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge (all P0)
|
||||
|
|
@ -362,14 +420,14 @@ int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
|
|||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect |
|
||||
| Requirements | 1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2 |
|
||||
| Requirements | 1.1-1.5, 2.1-2.3, 2.5, 3.1, 3.3, 5.1-5.2 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Replaces `xcecd_join_cluster`'s per-candidate loop (`controller_connector.c:237-311`) with: build `xce_client_config_t` via ConfigConverter once (candidates from `cc_ctx->cluster_cfg` when `--peers` given, else today's single-host fallback path unchanged per Req 1.5), then one `xcecli_ctx_init` + `xcecli_start` + `xcecli_join`.
|
||||
- Fixes the init-order bug by moving `cc_ctx->cluster_cfg` population (`main.c`) to before `xcecd_start()` is called, so the very first connect already sees the full candidate list (Req 1.4).
|
||||
- Replaces `xcecd_join_cluster`'s per-candidate loop (`controller_connector.c:239-312`) with: build ONE `xce_cluster_config_t` — from `cc_ctx->cluster_cfg` when `--peers` was given, or synthesized as a single-candidate `xce_cluster_config_t` (`candidates_count=1`, from the existing hardcoded `"localhost":9990` default, `main.c:333-334`) when it was not — run it through `xce_cluster_config_validate()`, then ConfigConverter, then one `xcecli_ctx_init` + `xcecli_start` + `xcecli_join`. **Deletes `init_client_config()`'s standalone direct-construction of `xce_client_config_t`** (`controller_connector.c:129-160`) and `xcecd_join_cluster()`'s separate "single-host fallback (original path)" branch (`controller_connector.c:297-311`) — both were a wholly parallel mechanism that never touched `xce_cluster_config_t`/`xce_parse_peers`/`xce_cluster_config_validate` at all (confirmed by source audit, 2026-08-10). Per the redefined Req 1 AC5, single-host is not a separately-maintained fallback; it is the `candidates_count=1` input to the exact same synthesize-validate-convert-connect sequence multi-candidate configs use.
|
||||
- Fixes the init-order bug by moving `cc_ctx->cluster_cfg` population (`main.c`) to before `xcecd_start()` is called, so the very first connect already sees the full candidate list (Req 1.4) — this fix composes naturally with the single-path unification above: since there is no longer a separate "first connect uses the single-host branch" code path to accidentally fall into, the only remaining ordering requirement is that whichever `xce_cluster_config_t` gets built (real `--peers` list or synthesized single-candidate) is fully assembled before the one synthesize-validate-convert-connect sequence runs, on both the first attempt and any later one.
|
||||
- Registers a rejoin callback (`xcecli_set_reconnect_handler`) that only sets a `needs_rejoin` flag and returns `0` — per "Rejoin callback constraint" above, it must not call `xcecli_join` (or any `xcecli_*` function) directly, since the library invokes it while `ac->failover.lock` is still held.
|
||||
- `checking_connection()`'s periodic `xcecli_hello` keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call `xcecd_join_cluster` on failure — a transport failure inside `xcecli_hello` already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each `xcecli_hello` call returns (lock now free), `checking_connection()` checks and clears `needs_rejoin`; if set, it calls `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate call — the same join payload used at startup.
|
||||
- Runs entirely on the existing dedicated connector thread (`start_controller_connector_thread`/`connector_to_controller`), preserving isolation from `hardware_monitor`/`system_monitor`/`process_manager` update paths (Req 3.1) — no new thread introduced.
|
||||
- `checking_connection()`'s periodic `xcecli_hello` keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call `xcecd_join_cluster` on failure — a transport failure inside `xcecli_hello` already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each `xcecli_hello` call returns (lock now free), `checking_connection()` checks `needs_rejoin`; if set, it calls `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate call — the same join payload used at startup. **The flag clears only on that join call's success** (Req 2 AC5): if `xcecli_join` itself returns `-E_XCE_NOT_READY` (its own retry budget exhausted), `needs_rejoin` stays set, so the next `xcecli_hello` tick's return retries the join — `checking_connection()`'s existing periodic cadence is what makes this retry naturally, no new timer needed.
|
||||
- Runs entirely on the existing dedicated connector thread (`start_controller_connector_thread`/`connector_to_controller`), preserving isolation from the main-loop `worker_process_update()` and `resource_sync_thread` liveness paths (Req 3.1) — no new thread introduced.
|
||||
|
||||
**Dependencies**
|
||||
- Inbound: `xcecd_start()` (P0)
|
||||
|
|
@ -379,7 +437,7 @@ int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
|
|||
|
||||
**Implementation Notes**
|
||||
- Integration: preserve existing WARN-level log statements around standby-redirect and candidate-exhaustion, now emitted from the library's classification path or immediately around the single init/join call, so Req 5 diagnostics are not lost in the simplification.
|
||||
- Validation: implementation must confirm (code read, not test-only) that `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` do not execute on the connector thread — flagged as a Validation Hook, not re-verified in this design pass.
|
||||
- Validation: implementation must confirm (code read, not test-only) that the connector thread is isolated from the main-loop `worker_process_update()` and the dedicated `resource_sync_thread` — the two recurring liveness paths — flagged as a Validation Hook, not re-verified in this design pass. (The three `*_update` calls previously cited — `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` — are one-shot init calls at `xcecd.c:476-480` on the main thread, not recurring liveness duties; corrected per 2026-08-10 gap re-validation.)
|
||||
- Risks: deleting the hand-rolled loop must not silently drop the `MAX_RETRY_COUNT`/`RETRY_INTERVAL` bounded-retry-then-`handle_disconnect()` behavior in `checking_connection()` — that outer bound (Req 2.3's "keep retrying" vs. any existing bounded-give-up behavior) must be reconciled explicitly during implementation, not left ambiguous.
|
||||
|
||||
### Scheduler
|
||||
|
|
@ -389,12 +447,14 @@ int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
|
|||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect |
|
||||
| Requirements | 1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2 |
|
||||
| Requirements | 1.1-1.3, 1.5-1.7, 2.1-2.3, 2.5, 3.2-3.3, 5.1-5.2, 7.1-7.6 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Replaces `sched_init`'s inline per-candidate loop (`scheduler.c:462-495`) with ConfigConverter + one `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` call.
|
||||
- Registers a rejoin callback that only sets a `needs_rejoin` flag and returns `0` (same constraint as worker — the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself). After each `xcecli_jobdispatch` call returns, the allocator checks and clears the flag; if set, it calls `xcecli_join(ctx, HostRole_scheduler, NULL)` as a separate call before continuing.
|
||||
- No new thread: `xcecli_jobdispatch` (the only backend call site, `scheduler.c:1289`) continues to run from the existing `sched_allocator` thread; blocking during a reconnect/backoff cycle is an accepted, benign degradation since the allocator has no useful work without a controller anyway, and `sched_job_picker` runs on its own separate thread unaffected (Req 3.1-3.2 satisfied by the existing thread split, not new code).
|
||||
- Replaces `sched_init`'s inline per-candidate loop (`scheduler.c:564-624`) with ConfigConverter + one `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` call. **Deletes `sched_init`'s "single-host path (original)" `else` branch** (`scheduler.c:613`, confirmed by source audit 2026-08-10 to be a wholly separate direct-construction mechanism when `--peers` is not given) — when no `--peers` is given, `sched_init` now synthesizes a single-candidate `xce_cluster_config_t` (`candidates_count=1`, from `init_config()`'s existing `"localhost":9990` default, `main.c:143-152`) and runs it through the identical `xce_cluster_config_validate()` → ConfigConverter → connect sequence as any real `--peers` list, per the redefined Req 1 AC5: single controller is `candidates_count=1` of the one mechanism, not a parallel fallback.
|
||||
- `--sim` mode early-returns before `sched_init()` (`main.c:296`), so the peers-parsing block only reaches `sched_init()` for normal-mode operation — the design's assumption that candidate-list wiring always runs before `sched_init()` in the normal-mode path is confirmed correct; only the line numbers shifted.
|
||||
- Registers a rejoin callback that only sets a `needs_rejoin` flag and returns `0` (same constraint as worker — the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself). **Corrected call site** (supersedes an earlier, incorrect claim that this runs on `sched_allocator`): `xcecli_jobdispatch` is called from `sched_publish_dispatch` (`scheduler.c:1257`), which is reached only via the decision-drain loop in `xce_sched_driver_run_normal_mode_round` (`driver.c:765-786`) — a loop that runs on the single main-loop thread (`main.c`'s `while(1)`, `main.c:348-349`), not on `sched_allocator`. `sched_allocator` only pushes decisions onto the `decisions` cbuf (`scheduler.c:336,387`); it never calls `xcecli_jobdispatch` directly. The check-`needs_rejoin`-then-`xcecli_join` call must therefore live in `sched_publish_dispatch`, immediately after its `xcecli_jobdispatch` call returns (lock released) — same thread, same call frame — **not** in `sched_allocator`. **The flag clears only once that `xcecli_join` call succeeds** (Req 2 AC5): on its own `-E_XCE_NOT_READY`, `needs_rejoin` stays set, so the next decision's publish-return retries the join — since Req 7's retry-until-success wrapper means `sched_publish_dispatch` returns (successfully) far more reliably now than before, this composes cleanly rather than needing its own separate retry loop.
|
||||
- **Corrected thread/backpressure model** (Critical Issue from 2026-08-10 design validation, resolved by narrowing below — supersedes the prior claim that `sched_job_picker` runs "unaffected"): the scheduler's real pipeline is three stages joined by two `cbuf`s: `sched_job_picker` → `jbuf` (capacity 1, `core.c:161` — the scheduler-core determinism invariant, never resizable without re-deriving `scheduler-core`'s Req 5 AC7) → `sched_allocator` → `decisions` (capacity `ndecisions`=5, `core.c:84`) → the main-loop thread's decision-drain loop → `xcecli_jobdispatch`. When that call blocks inside `XCECLI_FAILOVER_CALL`'s retry/backoff (lock held), the main thread stops draining `decisions`. `sched_allocator` can still produce more decisions before its own `cbuf_put(&core.decisions, ...)` blocks, and `sched_job_picker` can still produce jobs into `jbuf` for as long as allocator keeps draining it. So picker and allocator are **buffered, not isolated**: they continue briefly past the point the outage begins, then both stall until connectivity (and dispatch) resumes. **The exact size of that grace window is occupancy-dependent, not a fixed constant**: if `decisions` had Q items queued (1≤Q≤5) at the moment the main thread's pop grabs the decision that turns out to be un-publishable, allocator can push (6-Q) more decisions before it stalls, and picker's total new picks during the window come out to (7-Q) — ranging from 2 (Q=5, `decisions` was already near-full, common under load) up to 6 (Q=1, `decisions` was nearly empty). Earlier text in this section claimed "at least 6" as a guaranteed floor — that was backwards: 6 is the best-case ceiling (`ndecisions`+1), the guaranteed floor is as low as 1-2 depending on `jbuf`'s own occupancy at the same instant. This is a materially different, weaker guarantee than "unaffected," and a materially different, more accurate one than the original "at least 6" claim. This "stall until resumes" description is only actually true once the Req 7 fix below is in place — see that bullet for why, unfixed, the main thread can instead silently unstick itself by dropping a decision rather than genuinely staying stalled.
|
||||
- **Decision-publish retry-until-success-or-shutdown wrapper (Req 7, gap found in 2026-08-10 design validation)**: `xcecli_jobdispatch`/`xcecli_set_pend_reason` are `XCECLI_FAILOVER_CALL`-wrapped, which retries a *single call* up to a bounded budget (`max_retries`=20, backoff capped at `max_retry_ms`=5000ms — roughly 1-2 minutes total, `client_failover.c:281-282,312-317`) before returning `-E_XCE_NOT_READY`. Today's driver decision-drain loop (`driver.c:783-789`) does not inspect this return code — it logs and advances to the next queued decision regardless of outcome. Since `sched_commit_decision` already mutates scheduler-local job state (`job->allocs`, accounts) *before* the publish attempt (`scheduler.c` `sched_allocator`), a budget-exhausted publish today silently and permanently diverges scheduler-local state from what the controller was told — not a delay, a loss. Fix: `sched_publish_dispatch`/`sched_publish_pend_reason` (`scheduler.c:1257`/`1251`) wrap their `xcecli_*` call in `while (!sched->core.stop) { rc = XCECLI_FAILOVER_CALL(...); if (rc != -E_XCE_NOT_READY) break; WARN(extended-retry, ...); }` — checking `core.stop` **before** each attempt, not only after (a plain `do`/`while` checking the stop condition only at the bottom would let every already-queued decision behind the one currently stuck each burn one full ~100s budget cycle discovering shutdown was requested, compounding to minutes-per-queued-decision instead of a single bounded delay — caught in the 2026-08-10 design validation's third pass). Re-invoking the whole macro on each loop iteration resets its own internal retry state and starts a fresh budget, rather than adding a second retry mechanism. This is a straightforward extension of Req 2 AC3's existing "keep retrying, never permanently give up" language, applied per-decision instead of only at the connection level. **Shutdown interaction**: `stop_scheduler()`'s existing `cbuf_tryput(&core.decisions, NULL)` wake-up (`main.c:163`) only unblocks a thread parked in `cbuf_get(&core.decisions)` — it does nothing for a thread already inside this new retry loop, so the loop's own `core.stop` check is the only shutdown path. With the check moved to the top, only the ONE decision actively in-flight when shutdown is requested can cost up to one full budget cycle (~1-2 minutes); every decision still queued behind it in `decisions` sees `core.stop` already true and returns immediately without attempting a call, so total shutdown latency stays bounded to a single cycle regardless of queue depth — an accepted bounded tradeoff (same posture as scheduler-core's accepted 5s `OpCode_schedAdmin` blocking-wait case) since `libxce`'s retry macro itself is out of this spec's boundary to modify for a faster mid-attempt shutdown check. **Ordering**: decisions remain published strictly in `decisions` cbuf FIFO order (Req 7 AC3) because the drain loop still only advances to `cbuf_get()` the next decision after the current one's publish call returns — a stuck decision blocks the whole loop, it does not get skipped and revisited later. **Rejoin-check interaction**: the existing check-`needs_rejoin` step (Responsibilities bullet above) sits after this whole `while` loop, not after each inner attempt — it observes the loop's final, successful return, exactly the same as it would for a single non-retried call.
|
||||
|
||||
**Dependencies**
|
||||
- Inbound: `sched_init()` (P0)
|
||||
|
|
@ -404,6 +464,7 @@ int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
|
|||
|
||||
**Implementation Notes**
|
||||
- Integration: preserve the existing `WARN`/`INFO` log lines around candidate parsing (`main.c`) and add per-attempt/per-outcome logging around the single init/join call (today's loop logged each attempt; the library's own classification path should be checked for equivalent visibility, else add it locally).
|
||||
- **Req 3 AC2 scope note (Critical Issue from 2026-08-10 design validation, resolved 2026-08-10)**: the original merged AC1 wording ("shall continue performing ... allocation/decision processing ... without being blocked") was not met unconditionally by the scheduler — only for a bounded, occupancy-dependent grace window (floor as low as 1-2 jobs, ceiling `ndecisions`+1=6, see the corrected backpressure-model bullet above), after which `sched_allocator` and then `sched_job_picker` do block. `jbuf`'s capacity is a fixed invariant owned by the `scheduler-core` spec (out of this spec's boundary to change) and `decisions`' capacity is an existing config constant; redesigning either is out of scope here. Resolution: Requirement 3's single merged AC1 was split into a worker-only AC1 (unconditional, unchanged guarantee) and a scheduler-only AC2 (bounded — "continues briefly ... then blocks until connectivity is restored, at which point processing resumes automatically ... no job picked or decided during that bounded window is ever lost"), with the former heartbeat AC renumbered to AC3. `requirements.md` deliberately does not hardcode a specific job count — see the corrected backpressure-model bullet above for why that count is occupancy-dependent (floor 1-2, ceiling `ndecisions`+1=6), not a fixed constant an initial draft of this note incorrectly stated as a guaranteed "at least 6." `requirements.md` reflects this split and the corrected framing; the design and its bounded-buffering behavior now match the requirement it's held to, with Requirement 7 supplying the actual load-bearing guarantee (no loss, regardless of how many jobs the buffer absorbs).
|
||||
- Risks: today's total absence of reconnect logic means this is additive behavior change (scheduler previously required a manual restart after primary loss) — call out explicitly in the task/rollout notes so operators aren't surprised by scheduler now retrying indefinitely instead of exiting.
|
||||
|
||||
### apiserver
|
||||
|
|
@ -413,14 +474,14 @@ int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
|
|||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Single connect/join at startup using the full candidate list via the C library (through CGO), replacing the Go-level per-peer loop; config-syntax parity with scheduler/worker |
|
||||
| Requirements | 1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2 |
|
||||
| Requirements | 1.1-1.7, 2.4-2.5, 5.1-5.2, 6.1-6.2 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `RegisterToBackend()` (`clib.go`) is rewritten: build a C `xce_cluster_config_t` from `config.Peers` (or the single `Address`/`Port` fallback, Req 1.5) by calling `xce_parse_peers` via CGO (not the existing hand-written Go `parsePeer`), validate with `xce_cluster_config_validate`, convert with ConfigConverter, then call `xce_backend_init` **once** with the resulting `nmasters > 1` config — replacing the current `for _, peer := range config.Peers { tryConnect(...) }` Go loop entirely.
|
||||
- `RegisterToBackend()` (`clib.go`) is rewritten to build exactly ONE C `xce_cluster_config_t` regardless of source, then run it through the single synthesize-validate-convert-connect sequence worker/scheduler use: if `config.Peers` is non-empty, format it into the same `[name=]host:port[,...]` syntax and parse via `xce_parse_peers` (CGO, not the existing hand-written Go `parsePeer`); if empty, synthesize a one-candidate `xce_cluster_config_t` (`candidates_count=1`) directly from `config.Address`/`config.Port` — **either way** the result is validated with `xce_cluster_config_validate`, converted with ConfigConverter, then `xce_backend_init` is called **once** with the resulting config (`nmasters` in `[1, N_MAX_MASTERS]`, not necessarily `>1`). **Deletes `tryConnect()`'s direct `C.xce_client_config_t` construction** (`clib.go:79-101`) and the `for _, peer := range config.Peers { tryConnect(...) }` Go-level loop (`clib.go:116-135`) entirely, along with `parsePeer()` (`backend/peers.go:11`, retired per Req 6.1) — both `tryConnect()`'s Peers-loop call site and its no-Peers `Address`/`Port` call site (`clib.go:138`) were separate mechanisms that never touched `xce_cluster_config_t` at all (confirmed by source audit, 2026-08-10); per the redefined Req 1 AC5, a bare `Address`/`Port` config is `candidates_count=1` of the same mechanism, not a separate fallback.
|
||||
- `reconnect.go` holds two plain Go functions: (1) a `//export`-annotated flag-setter — registered as the library's rejoin callback, does no I/O, only sets a `needs_rejoin` atomic flag and returns `0` (per "Rejoin callback constraint" above: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join`/any CGO backend function itself); (2) a plain Go function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears `ApiserverBackendHealth`'s degraded state on success.
|
||||
- The flag-setter is **registered with the C library from `clib.go`'s own cgo preamble**, via a small static C shim (`xce_backend_rejoin_handler`, mirroring the existing `xce_backend_handler`/`go_push_handler` pattern already used for push dispatch) passed to `xcecli_set_reconnect_handler` — cgo symbol resolution is per-file (see `wire_dispatch_support.go`'s documented rationale for the identical `go_push_handler` wrapper), so the shim and the registration call must live in the same file as the `extern` declaration, not standalone in `reconnect.go`.
|
||||
- Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call `IsDegraded()` fail-fast check (ApiserverBackendHealth, below) and the post-call `needs_rejoin` check-and-clear are both implemented once, in a single shared internal helper each of the four `clib.go` wrappers calls around its `C.xcecli_*` invocation — not copy-pasted per wrapper.
|
||||
- **Idle-period reconnect gap (Req 2.4) fix**: `XCECLI_FF_CALL` — the macro backing `xcecli_heartbeat`, `heartbeat.go`'s existing periodic prober (`cmd/main.go:76`) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against `client_internal.h`/`client_failover.c`: `xcecli_invoke_rejoin` is called only from `xcecli_retry_next`, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: the heartbeat tick callback (`heartbeat.go`) tracks the previous tick's outcome; when a tick transitions from failure to success (transport just recovered), it explicitly calls `reconnect.go`'s rejoin function itself, then clears `ApiserverBackendHealth`. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code, just an explicit trigger point the write-call path already gets for free from the library.
|
||||
- Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call `IsDegraded()` fail-fast check (ApiserverBackendHealth, below) and the post-call `needs_rejoin` check are both implemented once, in a single shared internal helper each of the four `clib.go` wrappers calls around its `C.xcecli_*` invocation — not copy-pasted per wrapper. **The flag clears only once the follow-up `xcecli_join` call succeeds** (Req 2 AC5): if that call itself returns `-E_XCE_NOT_READY`, `needs_rejoin` stays set so the next backend-call wrapper's return retries the join, and `MarkHealthy()` is correspondingly only called on that eventual success — not on the mere attempt.
|
||||
- **Idle-period reconnect gap (Req 2.4) fix**: `XCECLI_FF_CALL` — the macro backing `xcecli_heartbeat`, `heartbeat.go`'s existing periodic prober (`cmd/main.go:76`) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against `client_internal.h`/`client_failover.c`: `xcecli_invoke_rejoin` is called only from `xcecli_retry_next`, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: on a failure→success transport transition, `heartbeat.go`'s tick callback sets the **same `needs_rejoin` atomic flag the write-path's C-side callback sets** — since `XCECLI_FF_CALL` never invokes that callback for this path, `heartbeat.go` sets the Go-side flag itself. On every subsequent tick where `xcecli_heartbeat` succeeds **and `needs_rejoin` is still true** (checked every tick, not only on the transition edge — Req 2 AC5 gap found in the 2026-08-10 design validation's second pass), it calls `reconnect.go`'s rejoin function; the flag (and `ApiserverBackendHealth`'s degraded state) clears only when that rejoin call itself succeeds. **Deliberately not gated on `ApiserverBackendHealth.IsDegraded()`** (a design-validation third-pass correction over an earlier draft of this fix): that state has its own independent TTL-based auto-clear (see ApiserverBackendHealth below) for request-path circuit-breaker purposes — if the TTL expired before the rejoin actually succeeded, gating the retry on "still degraded" would stop retrying the join even though it never succeeded, silently reintroducing the same starvation this fix exists to close. `needs_rejoin` and `ApiserverBackendHealth`'s degraded-until timestamp are two independent pieces of state serving two different purposes (one tracks "have I re-joined yet," the other "should request handlers fail fast right now") and must not be conflated. A transition-edge-only trigger would fire the rejoin attempt exactly once per outage and, if that one attempt's own `xcecli_join` hit its retry budget and failed, never retry — every subsequent tick would see success→success (transport already up) and never re-fire; checking the persistent flag instead of the transient edge makes each tick a natural retry opportunity for as long as apiserver remains un-joined. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code.
|
||||
|
||||
**Dependencies**
|
||||
- Inbound: `cmd/main.go` (P0)
|
||||
|
|
@ -432,8 +493,10 @@ int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
|
|||
```go
|
||||
// backend/clib.go
|
||||
// RegisterToBackend initialises the client using the full configured
|
||||
// candidate list (config.Peers, or the single Address/Port fallback) and
|
||||
// registers the rejoin callback. Single call — no per-peer Go loop.
|
||||
// candidate list — config.Peers if set, otherwise a single-candidate list
|
||||
// synthesized from config.Address/config.Port (not a separate code path,
|
||||
// see Req 1 AC5) — and registers the rejoin callback. Single call — no
|
||||
// per-peer Go loop.
|
||||
func RegisterToBackend(config *BackendConfig) error
|
||||
```
|
||||
- Preconditions: `StartNotificationListener` already called (unchanged from today).
|
||||
|
|
@ -470,6 +533,7 @@ func RegisterToBackend(config *BackendConfig) error
|
|||
**Implementation Notes**
|
||||
- Integration: TTL-based clearing (Risks & Mitigations in research.md) in addition to the rejoin-callback clear, so a missed/edge-case rejoin doesn't permanently wedge the flag.
|
||||
- Validation: unit tests cover degraded→healthy transitions via both the TTL expiry and explicit `MarkHealthy()`.
|
||||
- **Deliberately independent of `needs_rejoin`** (Req 2 AC5, correction from the 2026-08-10 design validation's third pass): `IsDegraded()`'s TTL-based auto-clear exists so request handlers eventually probe again even without an explicit `MarkHealthy()` — a legitimate circuit-breaker half-open pattern. It must **not** be used as the retry-trigger condition for the heartbeat-idle-path's rejoin attempts (ApiserverBackendBridge above) — the TTL could expire before an actual rejoin succeeds, and gating the rejoin retry on "still degraded" would then silently stop retrying the join while apiserver is still not actually joined. The rejoin-tracking `needs_rejoin` flag has no TTL and clears only on an actual `xcecli_join` success, independent of whatever `ApiserverBackendHealth`'s degraded-until timestamp is doing.
|
||||
|
||||
## Error Handling
|
||||
|
||||
|
|
@ -480,6 +544,7 @@ No new error taxonomy — reuse the existing controller status codes (`E_XCE_RED
|
|||
- **Malformed candidate-list config** (Req 1.6, 6.2): `xce_parse_peers`/`xce_cluster_config_validate` failure at startup → each component reports a clear message naming the offending entry and refuses to start (unchanged behavior for worker/scheduler; apiserver gains the same via the CGO route).
|
||||
- **All candidates unreachable at startup** (Req 1.6): `xcecli_start`/`xcecli_join` returns the last transport error after exhausting all candidates → component reports which candidates were attempted (Req 5.2) and fails startup (worker/scheduler: process exits as today; apiserver: `RegisterToBackend` returns an error, `cmd/main.go`'s existing handling applies unchanged).
|
||||
- **Runtime disconnect** (Req 2): transparently retried/rotated by the library; components only observe it through their own periodic calls' return codes and the rejoin callback — no new error category, just new logging around the existing outcomes.
|
||||
- **Scheduler decision-publish retry-budget exhaustion** (Req 7): `-E_XCE_NOT_READY` from `xcecli_jobdispatch`/`xcecli_set_pend_reason` is not treated as terminal — the caller re-invokes the same publish call rather than surfacing an error or dropping the decision (see SchedulerControllerClient's retry-until-success-or-shutdown wrapper). This is scheduler-specific: worker's periodic `xcecli_hello` carries no payload to lose on the same error, and apiserver's request/response calls correctly surface `-E_XCE_NOT_READY` to `ApiserverBackendHealth` since there the caller (the original HTTP client) is expected to retry, unlike scheduler's decisions which have no external retry source.
|
||||
- **Candidate count exceeds `N_MAX_MASTERS`**: ConfigConverter clamps and WARNs (not an error) — matches `xce_test`'s existing behavior exactly.
|
||||
- **apiserver backend degraded** (Req 4.1): mapped to a clear 503-style API error, distinct from other 5xx causes, so operators/clients can distinguish "backend down, retrying" from an unrelated internal error.
|
||||
|
||||
|
|
@ -489,11 +554,15 @@ Per-candidate connect/reconnect attempts and outcomes are logged (Req 5) using e
|
|||
## Testing Strategy
|
||||
|
||||
- **Unit Tests**:
|
||||
- `ConfigConverter`: clamps at `N_MAX_MASTERS` with WARN when given more candidates; preserves candidate order; rejects `NULL`/empty input (`test_cluster.c`).
|
||||
- `ConfigConverter`: clamps at `N_MAX_MASTERS` with WARN when given more candidates; preserves candidate order; rejects `NULL`/empty input (`test_cluster.c`); a `candidates_count=1` input produces `nmasters=1` through the identical code path as any other count (Req 1.5) — no branch, no special case.
|
||||
- `N_MAX_MASTERS == MAX_HA_CANDIDATES` (Req 1.7): a `_Static_assert` (or equivalent compile-time check) in `client.h` or a `test_cluster.c` case asserting the two constants stay equal — a regression guard against reintroducing the divergence this design pass reconciled. Also re-run `test_libxce_retry.c`'s `fo_make_client_masters()` fixture with its corrected slot count and confirm it doesn't silently exercise an uninitialized `masters[3]`.
|
||||
- apiserver `health.go`: degraded→healthy via `MarkHealthy()`, via TTL expiry; concurrent goroutine safety (`health_test.go`).
|
||||
- apiserver CGO-routed parser: same malformed-input cases scheduler/worker already reject, now asserted through the Go call site (`clib_test.go`).
|
||||
- Rejoin flag-setter callbacks (worker, scheduler, apiserver): with a real `ac->failover.lock` held (simulating the library's call context per "Rejoin callback constraint"), invoking the registered callback returns promptly without blocking and performs no I/O — a regression guard against reintroducing the self-deadlock this design pass found.
|
||||
- Rejoin retry-on-failure (Req 2 AC5, worker/scheduler/apiserver): using a test double for `xcecli_join` that fails N times before succeeding, assert the pending-rejoin indicator (flag or degraded-state) stays set/degraded after each failed attempt and is cleared only on the eventual success; assert the next naturally-occurring opportunity (next `xcecli_hello`/decision-publish/backend-call return, or next heartbeat tick for apiserver's idle path) is what re-attempts it, not a dedicated new retry loop — a regression guard against the transition-edge/clear-before-success versions of this bug found in the 2026-08-10 design validation's second pass.
|
||||
- **Integration Tests**:
|
||||
- Scheduler decision-publish redelivery (Req 7): using a test double for `xcecli_jobdispatch`/`xcecli_set_pend_reason` that returns `-E_XCE_NOT_READY` for N calls before succeeding, assert the same decision is retried (not dropped, not skipped) and eventually published; assert decisions queued behind it publish afterward in original order (Req 7 AC1-3); assert setting `sched->core.stop` mid-retry breaks the loop within one budget cycle (Req 7 AC2); assert a WARN-level log line distinguishes extended-retry success from first-attempt success (Req 7 AC4); with multiple decisions queued (`decisions` at or near its 5-item capacity) when `core.stop` is set mid-retry on the first, assert total shutdown latency stays bounded to roughly one budget cycle, not one per queued decision — a regression guard against the compounding-shutdown-latency bug found in the 2026-08-10 design validation's third pass.
|
||||
- Unified single-host path (Req 1.5): starting each of worker, scheduler, and apiserver with no `--peers`/empty `backend.peers` reaches the identical `xce_cluster_config_validate` → ConfigConverter → connect call sequence as a single-entry `--peers` config — assert via the same log-line signature both paths now share (there is no longer a distinct "single-host mode" log message), not just "it connects."
|
||||
- Worker/scheduler startup against a 3-candidate config where candidate[0] is a live standby — assert the component joins as expected without operator intervention (Req 1.1-1.3), extending each component's `testsuite/`.
|
||||
- Worker/scheduler reconnect: kill the currently-joined primary mid-session, assert the component reconnects and re-joins the new primary without restart (Req 2.1-2.3), reusing the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern already used for controller HA soak testing — this is also the test that would hang (not fail cleanly) if the self-deadlock were reintroduced, so it must run with a wall-clock timeout that fails loud rather than hanging CI.
|
||||
- **E2E Tests**:
|
||||
|
|
|
|||
|
|
@ -13,22 +13,8 @@ Scope: per-component master-list config + retry policy + liveness-safe blocking
|
|||
|
||||
None of the three production daemons match this pattern today; each has a different gap:
|
||||
|
||||
- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:610`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:127-158`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:237-293`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
|
||||
- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:124-156`, `scheduler.c:462-495`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
|
||||
- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only.
|
||||
|
||||
In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.
|
||||
|
||||
A separate, partially-built shared primitive already exists in libxce (`src/include/xce/cluster.h`: `xce_connect_cluster()`, `xce_reconnect_init/step/fini()`, backed by `xce_parse_peers()`/`xce_cluster_config_validate()`) — but its production connect path (`do_connect()` in `src/lib/cluster.c`) is a stub that always returns `NULL` outside unit tests, and it operates on a raw `endpoint_t*`, not the higher-level `xcecli_ctx_t` the daemons actually use for the join/request/reply protocol. No production code calls it. Whether the design phase completes/adopts this primitive or builds directly on `xce_client_config_t.nmasters` (the `libxce-failover`/`client_failover.c` path xce_test already uses) is a design decision, not a requirements one — both existing mechanisms are noted here as prior art this feature must reconcile with, not duplicate.
|
||||
|
||||
### Background (verification findings, 2026-07-19)
|
||||
|
||||
`xce_test`'s `masters` setting (`src/controller/testsuite/test.c:345-377`) is the reference implementation: it parses a `host:port,...` list via `xce_parse_peers()`, sets `config.nmasters = candidates_count`, fills the full `config.masters[]` array, and calls `xcecli_ctx_init()` once — so libxce's own `client_failover.c` EndpointRotator handles mid-session failover across all candidates.
|
||||
|
||||
None of the three production daemons match this pattern; each has a different gap:
|
||||
|
||||
- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:610`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:127-158`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:237-293`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
|
||||
- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:124-156`, `scheduler.c:462-495`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
|
||||
- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:537`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:129-160`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:239-312`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
|
||||
- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:299-324`, `scheduler.c:564-624`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
|
||||
- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only.
|
||||
|
||||
In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.
|
||||
|
|
@ -36,7 +22,7 @@ In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per co
|
|||
## Boundary Context
|
||||
|
||||
- **In scope**: scheduler, worker, and apiserver each using their *full* configured controller-candidate list (not just the first entry) for both the initial connection and any later reconnect; automatic reconnect across candidates for all three components (today scheduler and apiserver have none); liveness-safe retry/backoff on the worker and scheduler so a slow/unreachable candidate never stalls their other liveness-critical duties; consistent candidate-list configuration syntax across all three components; operator-visible diagnostics for connect/reconnect attempts and failures; apiserver behavior toward its own API clients while its backend connection is down or being re-established.
|
||||
- **Out of scope**: `xce_test` (already the correct reference client, not a target of this feature); raising the validated 3-candidate limit; controller-side changes (peer authentication, `validate_host()`, redirect-reply population — already shipped by `libxce-failover`); TLS/certificate distribution mechanics beyond the existing CA-cert configuration; the lightweight session-based rejoin/re-attach optimization (separate backlog item); recovering worker job-state that was lost during a disconnect window (owned by the separate `worker-rejoin-resync` item — this feature restores the *connection*, not job-state content).
|
||||
- **Out of scope**: `xce_test` (already the correct reference client, not a target of this feature); controller-side changes (peer authentication, `validate_host()`, redirect-reply population — already shipped by `libxce-failover`); TLS/certificate distribution mechanics beyond the existing CA-cert configuration; the lightweight session-based rejoin/re-attach optimization (separate backlog item); recovering worker job-state that was lost during a disconnect window (owned by the separate `worker-rejoin-resync` item — this feature restores the *connection*, not job-state content). Reconciling the candidate-count limit (`N_MAX_MASTERS` deriving from `config.h`'s `MAX_HA_CANDIDATES`, Req 1 AC7) is now explicitly **in** scope — see Requirement 1; an earlier draft of this document listed raising it as out of scope, superseded 2026-08-10.
|
||||
- **Adjacent expectations**: depends on `libxce-failover` (shipped) for the underlying retry primitive and controller status taxonomy (redirect vs. standby vs. not-ready); does not duplicate or replace that mechanism, only ensures each component actually drives it with a complete candidate list.
|
||||
|
||||
## Requirements
|
||||
|
|
@ -49,8 +35,9 @@ In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per co
|
|||
2. If the first attempted candidate is unreachable, the component shall proceed to the next configured candidate without operator intervention or a restart.
|
||||
3. If a configured candidate responds as standby rather than primary, the component shall treat that as informational and proceed to the next configured candidate, not as a fatal connection error.
|
||||
4. The worker component shall use the fully configured candidate list on its first connection attempt at startup, not only on subsequent reconnects.
|
||||
5. Where only one candidate (or none) is configured, the component shall behave as it does today in single-controller mode.
|
||||
5. Where only one candidate is configured — whether from a single-entry `--peers`/`backend.peers` list, or synthesized from the legacy single-host default when no candidate list is given at all — the component shall treat it as a one-element candidate list and drive it through the exact same validate/convert/connect pipeline used for a multi-candidate configuration. A single controller is not a separately-maintained code path or a limit case of "up to 3 controllers" — it is the `candidates_count=1` case of the one HA-candidate mechanism, and observable behavior (log lines, error handling, retry/reconnect semantics) shall be identical in shape to the multi-candidate case, just with one candidate.
|
||||
6. If every configured candidate is unreachable or rejects the join at startup, then the component shall report a clear startup failure identifying which candidates were attempted.
|
||||
7. The maximum number of controller candidates a component can accept shall be defined by a single project-wide constant (`MAX_HA_CANDIDATES` in `src/include/config.h`), not independently duplicated or re-defined elsewhere. `N_MAX_MASTERS` (`src/include/client.h`) shall derive its value from `MAX_HA_CANDIDATES` rather than being a separately-maintained constant that can silently diverge from it, as it does today (3 vs. 4).
|
||||
|
||||
### Requirement 2: Automatic reconnect after controller failover
|
||||
**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to automatically reconnect to a surviving controller candidate when their active controller connection is lost, so that a controller failover does not require manually restarting these daemons.
|
||||
|
|
@ -60,13 +47,15 @@ In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per co
|
|||
2. While reconnecting, the component shall keep retrying across the full configured candidate list until one candidate accepts the connection as primary or the component is shut down by an operator.
|
||||
3. If a reconnect sweep exhausts all configured candidates without success, then the component shall continue retrying rather than permanently giving up, so the component self-heals once a primary becomes reachable again.
|
||||
4. When apiserver reconnects to a new primary after a controller failover, apiserver shall resume receiving backend push notifications without requiring an apiserver process restart.
|
||||
5. If the application-level re-join call issued after a successful transport-level reconnect itself fails (including its own retry-budget exhaustion), the component shall retry that re-join rather than treat the failure as final; a pending-rejoin indicator shall be cleared only once the re-join call succeeds, not merely once it has been attempted, so a component is never left transport-connected but silently un-joined to the cluster.
|
||||
|
||||
### Requirement 3: Reconnect retry does not block other liveness-critical work
|
||||
**Objective:** As an HA cluster operator, I want the worker and scheduler's controller-reconnect retries to never block their other liveness-critical processing, so that a slow or unreachable controller candidate does not make the daemon itself appear unresponsive.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. While a worker or scheduler is retrying a controller connection (including any backoff delay between attempts), the component shall continue performing its other liveness-critical duties (worker: hardware/process monitoring and job-state tracking; scheduler: its allocation/decision processing) without being blocked by the retry attempt or backoff delay.
|
||||
2. The worker and scheduler's reconnect retries shall not delay or prevent the component's own heartbeat responses to a connected controller, so a stalled reconnect attempt elsewhere cannot itself trigger an unnecessary failover.
|
||||
1. While a worker is retrying a controller connection (including any backoff delay between attempts), the worker shall continue performing its other liveness-critical duties (hardware/process monitoring and job-state tracking) without being blocked by the retry attempt or backoff delay.
|
||||
2. While a scheduler is retrying a controller connection (including any backoff delay between attempts), the scheduler's allocation/decision processing shall continue briefly past the point the retry begins, then block until connectivity is restored, at which point processing shall resume automatically from exactly where it left off. This guarantee is bounded, not indefinite: the scheduler's picker and allocator threads are buffered against a blocked retry only by the depth of its internal `jbuf`/`decisions` queues — a `scheduler-core` determinism invariant (`jbuf` capacity fixed at 1) that this feature does not adjust — so allocation/decision processing does stall for the remainder of an outage once that buffer is exhausted. The exact number of jobs this buffering absorbs before stalling depends on buffer occupancy at the moment the outage begins (as few as 1-2, as many as `ndecisions`+1=6 under today's buffer sizes) — an implementation detail of `scheduler-core`, not a fixed guarantee this requirement makes. What this requirement does guarantee is that no job picked or decided during that bounded window is ever lost, whether it is still buffered when the outage begins or blocks mid-publish — see Requirement 7.
|
||||
3. The worker and scheduler's reconnect retries shall not delay or prevent the component's own heartbeat responses to a connected controller, so a stalled reconnect attempt elsewhere cannot itself trigger an unnecessary failover.
|
||||
|
||||
### Requirement 4: apiserver behavior toward API clients during backend disconnection
|
||||
**Objective:** As an API client of apiserver, I want a clear, immediate response when the backend controller connection is down, so that I am not left waiting indefinitely for requests that cannot currently be served.
|
||||
|
|
@ -88,3 +77,17 @@ In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per co
|
|||
#### Acceptance Criteria
|
||||
1. The candidate-list configuration accepted by apiserver shall use the same `[name=]host:port[,...]` syntax already accepted by scheduler and worker.
|
||||
2. If a component's candidate-list configuration is malformed, then the component shall reject it with an error identifying the malformed entry, consistent with how the other components report malformed candidate-list configuration.
|
||||
|
||||
### Requirement 7: Scheduler decision delivery survives extended outages without loss
|
||||
**Objective:** As an HA cluster operator, I want the scheduler to never silently lose a job dispatch or pend decision because the controller was unreachable longer than a single retry budget, so that scheduler-local job state and controller-observed job state cannot permanently diverge after a failover.
|
||||
|
||||
#### Background
|
||||
`xcecli_jobdispatch`/`xcecli_set_pend_reason` are wrapped in `XCECLI_FAILOVER_CALL`, which retries a *single call* up to a bounded budget (`max_retries`, default 20; backoff capped at `max_retry_ms`, default 5000ms — roughly 1-2 minutes total) before returning `-E_XCE_NOT_READY` (`client_failover.c:281-282,312-317`). This is a per-call budget, not indefinite retry. Today, the scheduler's decision-drain loop (`driver.c`'s decision-drain loop calling `xce_sched_driver_on_decision`) does not inspect this return code — it logs and moves on to the next queued decision regardless of outcome. Because `sched_commit_decision` already mutates scheduler-local job state (`job->allocs`, accounts) *before* the publish attempt (`scheduler.c` `sched_allocator`), a budget-exhausted publish leaves the scheduler believing a job is dispatched/pending while the controller was never told — a silent, permanent state divergence, not a delay.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When publishing a decision (`xcecli_jobdispatch` or `xcecli_set_pend_reason`) returns `-E_XCE_NOT_READY` (single-call retry budget exhausted) while the scheduler has not been shut down, the scheduler shall retry publishing that same decision rather than discarding it or advancing to the next queued decision.
|
||||
2. The scheduler shall keep retrying a given decision's publish across repeated retry-budget exhaustion until either the publish succeeds or the scheduler is shut down by an operator, consistent with Requirement 2 AC3's "never permanently give up" guarantee applied at the level of an individual decision, not only the connection.
|
||||
3. Decisions already queued behind a stuck decision shall not be published out of order — the scheduler's existing decision-ordering guarantee is preserved while a decision is being retried.
|
||||
4. When a decision publish enters extended retry (one or more budget exhaustions) and when it eventually succeeds, the scheduler shall log this distinctly from an ordinary single-attempt success, so operators can distinguish "delayed but delivered" from "delivered on the first try."
|
||||
5. This requirement applies to the scheduler's own decision-publishing direction only; it does not extend to worker job-state resync on rejoin (owned by the separate `worker-rejoin-resync` backlog item) or to any controller-side redelivery mechanism.
|
||||
6. This requirement guarantees delivery only for as long as the scheduler process itself stays running through the outage — decisions held in the in-memory `decisions`/`jbuf` queues (including the one actively being retried) do not survive a scheduler process crash or restart, since no persistence mechanism for queued-but-unpublished decisions exists today or is added by this requirement. A scheduler restart during an outage loses the same in-flight decisions it would have lost before this requirement, and requires a fresh scheduling pass after reconnecting — this requirement closes the silent-drop-while-still-running gap, not a durability gap.
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@
|
|||
|
||||
### Does the library mechanism, once enabled, also cover runtime reconnect (Requirement 2) automatically?
|
||||
- **Findings**: Yes for any component whose normal operation already calls `xcecli_*` functions periodically or on demand — every one of those calls is wrapped in `XCECLI_FAILOVER_CALL`/`XCECLI_FAILOVER_CALL_COUNT`, so a transport failure on any call transparently triggers rotate+reconnect+retry within that same call. Fire-and-forget calls (`xcecli_nop`, `xcecli_heartbeat`) use `XCECLI_FF_CALL` instead: a bounded single best-effort reconnect attempt, never the full retry budget — this is the library's own liveness-safety design for exactly the concern in Requirement 3.
|
||||
- **Implications**: Scheduler's `xcecli_jobdispatch` calls (scheduler.c:1289, called from the `sched_allocator` thread) and apiserver's per-request backend calls (`gCtx.ctx` in clib.go) get transparent reconnect for free once `nmasters` is set correctly — no new polling/reconnect loop needs to be written for either component.
|
||||
- **Implications**: Scheduler's `xcecli_jobdispatch` calls (scheduler.c:1259, called from `sched_publish_dispatch` on the main driver-loop thread — see "Thread-affinity" correction below, not `sched_allocator`) and apiserver's per-request backend calls (`gCtx.ctx` in clib.go) get transparent reconnect for free once `nmasters` is set correctly — no new polling/reconnect loop needs to be written for either component.
|
||||
|
||||
### Does anything need a NEW re-join after a transparent reconnect, beyond what the library does automatically?
|
||||
- **Findings**: The library's rejoin hook exists precisely because reconnecting the transport does not re-establish application-level cluster membership — a new primary does not automatically know a worker/scheduler/apiserver rejoined unless it re-sends its join (role) message. `xcecli_reconnect_cb` "runs on the calling thread" and returns 0 to resume the interrupted op or <0 to treat the rejoin as failed (loop rotates/backs off again).
|
||||
|
|
@ -27,13 +27,13 @@
|
|||
|
||||
### Thread-affinity of blocking retry calls (Requirement 3)
|
||||
- **Findings**: Worker already isolates all controller-connection work (`controller_connector_start`, `xcecd_join_cluster`, and the periodic `checking_connection()` health-check) onto one dedicated thread (`start_controller_connector_thread` / `connector_to_controller`), separate from `hardware_monitor`/`system_monitor`/`process_manager` update calls in `xcecd_start()`. This isolation already satisfies Requirement 3 structurally for the worker — it is not something this feature needs to newly build, only preserve while removing the redundant candidate loop.
|
||||
- Scheduler already runs `sched_job_picker` and `sched_allocator` as two separate `pthread_create`d threads (scheduler.c:511,516); `xcecli_jobdispatch` (the only call site found that could block on retry, scheduler.c:1289) executes from the allocator thread only. A blocked allocator during a controller outage is a benign, expected degradation (there is no work the allocator can do without a controller anyway) and does not stall the job picker or any liveness signal scheduler sends elsewhere — scheduler has no independent heartbeat-carrying thread to protect. No new threading is needed for the scheduler.
|
||||
- **Corrected 2026-08-10**: `xcecli_jobdispatch` (`scheduler.c:1259`) does NOT execute from `sched_allocator`. It is called from `sched_publish_dispatch`, reached only via the decision-drain loop inside `xce_sched_driver_run_normal_mode_round` (`driver.c:765-786`), which runs on the single main-loop thread (`main.c`'s `while(1)`, `main.c:348-349`). `sched_allocator` (`scheduler.c:295-392`) only does `cbuf_get(&core.jbuf, ...)` then `cbuf_put(&core.decisions, ...)` — it never calls `xcecli_jobdispatch` itself. The scheduler's real pipeline is three stages joined by two `cbuf`s: `sched_job_picker` → `jbuf` (capacity 1, `core.c:161`, the scheduler-core determinism invariant) → `sched_allocator` → `decisions` (capacity `ndecisions`=5, `core.c:84`) → the main-loop thread → `xcecli_jobdispatch`. When that call blocks in retry/backoff, the main thread stops draining `decisions`; `sched_allocator` and `sched_job_picker` can still produce a few more jobs before their own `cbuf_put` calls block. So the two threads are **buffered for a bounded, occupancy-dependent grace window, not indefinitely isolated** — the earlier "benign, does not stall the job picker" claim was wrong; both threads do eventually stall for the remainder of an outage. **Corrected 2026-08-10 (second pass)**: the grace window's size depends on how full `decisions` was at the moment of the stall (guaranteed floor as low as 1-2 jobs, ceiling `ndecisions`+1=6) — an earlier version of this note claimed "~6-job" as if it were the guaranteed size; that was the best-case ceiling, not a floor. See design.md's SchedulerControllerClient section for the full derivation, the Req 3 AC1→AC2 scope-narrowing, and Requirement 7 (the actual load-bearing guarantee: no job lost regardless of how many the buffer absorbs).
|
||||
- apiserver: unlike the C daemons, ALL request-handling goroutines that touch the backend share the same `gCtx.ctx`, serialized by the library's per-context `failover.lock`. Under a real outage this could queue every concurrent API request behind the full retry budget, which is a client-experience problem (Requirement 4), not a cluster-liveness problem. Needs a lightweight fail-fast guard in the Go layer — see Design Decisions.
|
||||
- **Implications**: No new dedicated thread is required for worker or scheduler (existing isolation already suffices — flagged as a Validation Hook to confirm during implementation, not re-derived here). apiserver needs a new, small piece of state (not a thread) to short-circuit blocking calls when the backend is known-down.
|
||||
- **Implications**: No new dedicated thread is required for worker (existing isolation already suffices). Scheduler needs no new thread either, but Req 3 AC1 needs its wording narrowed to match the bounded (not indefinite) buffering the existing `jbuf`/`decisions` cbufs actually provide — see design.md Critical Issue note. apiserver needs a new, small piece of state (not a thread) to short-circuit blocking calls when the backend is known-down.
|
||||
|
||||
### Configuration surface mismatch: `N_MAX_MASTERS` vs `MAX_HA_CANDIDATES`
|
||||
- **Findings**: `src/include/client.h:28` defines `N_MAX_MASTERS (3)` — the hard cap on `xce_client_config_t.masters[]`. `src/include/config.h:78` defines `MAX_HA_CANDIDATES 4` — the cap on `xce_cluster_config_t.candidates[]` used by `xce_parse_peers`. These are different constants. `xce_test` (`test.c:360-367`) already handles the mismatch correctly: parse up to 4 candidates, then clamp to `N_MAX_MASTERS` (3) with a `WARN` if truncated.
|
||||
- **Implications**: Any new shared conversion helper must replicate this exact clamp-and-warn behavior (not silently drop entries, not error out) so all components behave identically when 4 candidates are configured. Reconciling the two constants into one is out of scope (pre-existing, unrelated to this feature).
|
||||
- **Implications**: Any new shared conversion helper must replicate this exact clamp-and-warn behavior (not silently drop entries, not error out) so all components behave identically when 4 candidates are configured. **Scope expanded 2026-08-10 (explicit user direction)**: reconciling the two constants is now IN scope — `N_MAX_MASTERS` is redefined to derive from `MAX_HA_CANDIDATES` (`config.h` becomes the single source of truth, Req 1 AC7), raising it 3→4. Source audit confirmed this is safe: no wire-protocol array assumes 3 (`masters[]` is never serialized), no header-include cycle (`config.h` depends only on `<stdint.h>`), and `xce_test`'s clamp already references `N_MAX_MASTERS` symbolically so it absorbs the raise automatically. Two latent bugs found by this audit that the raise would otherwise silently trip: `src/worker/src/main.c:414` and `src/scheduler/src/main.c:314` both hardcode a literal `> 3` check instead of referencing the macro — need fixing to `> N_MAX_MASTERS` alongside the raise. `src/lib/testsuite/test_libxce_retry.c`'s `fo_make_client_masters()` fixture also hardcodes exactly 3 literal host/port entries then copies `N_MAX_MASTERS` slots — needs a 4th literal added or it silently copies an uninitialized slot post-raise.
|
||||
|
||||
### TLS/CA-cert wiring gap (pre-existing, out of boundary)
|
||||
- **Findings**: `xce_client_config_t` (the struct actually consumed by `xcecli_ctx_init`/`xcecli_start`, i.e. the real connection path) has no CA-cert field at all — only `nmasters`/`masters`/`master_ports`/`from`/`key`. The `ca_cert_path` that worker/scheduler's `--ca-cert` flag populates on `xce_cluster_config_t` is validated by `xce_cluster_config_validate()` but is only ever read by the orphaned `xce_connect_cluster()` stub, never by the real connect path.
|
||||
|
|
@ -81,7 +81,7 @@
|
|||
### Decision: rejoin callbacks must be lock-free flag-setters, never direct `xcecli_join` calls
|
||||
- **Context**: A second `/kiro-validate-design` pass, cross-checking the library-invoked rejoin path against `client_failover.c` directly rather than only its header comments, found that `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` — a plain, non-recursive `pthread_mutex_t` (`mylock_init` → `pthread_mutex_init(l, NULL)`) — for its entire retry loop, and `xcecli_invoke_rejoin()` (which calls the registered rejoin callback) runs synchronously *inside* that loop, lock held (`client_failover.c:397`). Every component's original design had its rejoin callback call `xcecli_join(...)` directly — itself `XCECLI_FAILOVER_CALL`-wrapped, so it would try to re-lock the same mutex on the same thread: a guaranteed self-deadlock on the very first real failover. This was never caught because `xcecli_set_reconnect_handler` has zero production callers and the existing unit test's callback never calls back into the client (it's a no-op probe).
|
||||
- **Alternatives Considered**: (1) Have the callback call `xcecli_join` directly (original design — deadlocks). (2) Expose a new, lock-free/raw join primitive from libxce for rejoin callbacks to use (would require modifying `libxce-failover`, violating this spec's Allowed Dependencies boundary of consuming it read-only). (3) Callback only sets a lightweight flag and returns immediately (no I/O, no lock); the daemon's own calling code checks and clears the flag *after* the triggering call returns (lock free by then) and issues the real `xcecli_join` as a separate call.
|
||||
- **Selected Approach**: (3), applied uniformly to worker (`checking_connection`), scheduler (the allocator, after `xcecli_jobdispatch`), and apiserver (each backend-call wrapper, after its `C.xcecli_*` call). apiserver's heartbeat-triggered path (Task 4.4) was already structured this way by construction (it acts after `xcecli_heartbeat` returns) and needed no change.
|
||||
- **Selected Approach**: (3), applied uniformly to worker (`checking_connection`), scheduler (`sched_publish_dispatch`, the main-loop thread's actual `xcecli_jobdispatch` call site — not the allocator, corrected 2026-08-10), and apiserver (each backend-call wrapper, after its `C.xcecli_*` call). apiserver's heartbeat-triggered path (Task 4.4) was already structured this way by construction (it acts after `xcecli_heartbeat` returns) and needed no change.
|
||||
- **Rationale**: Fixes the deadlock without touching `libxce` at all, staying inside this spec's stated boundary; the fix is a small, uniform pattern applied identically across all three components rather than a special case.
|
||||
- **Trade-offs**: One extra check-and-clear step after each relevant call site, in exchange for correctness; negligible cost.
|
||||
- **Follow-up**: Worth flagging upstream to `libxce-failover`'s owners that the rejoin-hook contract (`client.h`'s doc comment for `xcecli_reconnect_cb`) doesn't document this lock-held constraint — a future consumer without this spec's context could reintroduce the deadlock. Not fixed here (out of boundary), but the regression-guard unit test added in design.md's Testing Strategy exists specifically because of this gap in the upstream contract's documentation.
|
||||
|
|
@ -108,6 +108,56 @@
|
|||
- **Rationale**: Out of this feature's boundary — removing dead code is a separate hardening decision, not required to satisfy any requirement here, and touching it risks scope creep into an unrelated cleanup.
|
||||
- **Follow-up**: Worth a future hardening backlog note (superseded/dead code) — not filed as part of this spec.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Validation (2026-08-10)
|
||||
|
||||
Re-verified all code citations against the current `scheduler-framework` branch (post `scheduler-core` refactor, post worker CLI cleanup commits `9ffb1d5`–`0df1cf5`).
|
||||
|
||||
### Worker (xcecd)
|
||||
|
||||
**CLI changes orthogonal to spec.** The 6 worker commits removed `-c`/`-p`/`-m` flags, config-file loading, `xcecd_conf.c/h`, and `worker.yaml`. Added `-v` verbose logging, fixed `logdir` default and `-h` ordering. None touched `controller_connector.c`/`controller_connector.h` — the failover-relevant code is unchanged.
|
||||
|
||||
**Bug-1510 confirmed still present.** `cc_ctx->cluster_cfg = cluster_cfg` at current `main.c:537` still runs AFTER `xcecd_start()` at `main.c:531`. `controller_connector_start()` at `xcecd.c:515` performs the first join via `init_client_config()` (`controller_connector.c:129-160`, `nmasters=1` at line 144) before `cluster_cfg` is populated. Spec tasks 2.1–2.3 correctly target this.
|
||||
|
||||
**Line drift (minor):**
|
||||
| spec cite | current |
|
||||
|-----------|---------|
|
||||
| `main.c:610` | `main.c:537` |
|
||||
| `controller_connector.c:127-158` | `controller_connector.c:129-160` |
|
||||
| `controller_connector.c:237-311` | `controller_connector.c:239-312` |
|
||||
|
||||
**Task 2.3 validation hook stale.** Task says to verify `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` don't run on connector thread. These three are one-shot init calls at `xcecd.c:476-480` — never called again. The actual recurring liveness duties are `worker_process_update()` (main loop, line 577) and `resource_sync_thread` (dedicated thread, `resource_sync.c:293-330`). The architectural property (connector thread isolated) IS correct, but the named evidence is wrong. Recommend rewording to reference main loop/`resource_sync_thread`.
|
||||
|
||||
### Scheduler
|
||||
|
||||
**HA candidate loop untouched.** `scheduler.c:564-624` (was ~462-495 at spec-write). Loop logic unchanged; only line numbers shifted by `scheduler-core` refactor (doc comments, `core.c` extraction, `--sim` CLI additions in `main.c`).
|
||||
|
||||
**No-reconnect claim confirmed.** No `xcecli_set_reconnect_handler` call in any scheduler file. No reconnect path after `sched_init()`.
|
||||
|
||||
**Line drift (minor):**
|
||||
| spec cite | current |
|
||||
|-----------|---------|
|
||||
| `scheduler.c:462-495` | `scheduler.c:564-624` |
|
||||
| `scheduler.c:1289` (`xcecli_jobdispatch`) | `scheduler.c:1259` |
|
||||
| `scheduler.c:511,516` (pthread_create) | `scheduler.c:626,632` |
|
||||
| research.md L22: `scheduler.c:1289` | `scheduler.c:1259` |
|
||||
| research.md L30: `scheduler.c:511,516` | `scheduler.c:626,632` |
|
||||
|
||||
**New structural note for design:** `main.c` now early-returns for `--sim` mode (line 296), inserted between peers-parsing block and `sched_init()`. Doesn't affect normal-mode path — peers block still runs before `sched_init()` for non-`--sim` — but design should note this guard exists.
|
||||
|
||||
### Apiserver
|
||||
|
||||
Not re-verified in depth (no flagged changes in this area). Light sanity check confirms `RegisterToBackend` still loops `config.Peers` with per-peer `tryConnect`.
|
||||
|
||||
### Requirements.md pre-existing issues
|
||||
|
||||
**Duplicate Background section.** Lines 10-22 and 24-34 are byte-identical — copy-paste artifact at spec-creation time. Should deduplicate.
|
||||
|
||||
### Summary
|
||||
|
||||
All architectural findings and target bugs in the spec remain substantively accurate. Only citation line numbers and task 2.3 wording need refreshing. No design decisions invalidated. Spec is ready for implementation.
|
||||
|
||||
## Risks & Mitigations
|
||||
- Deleting worker's/scheduler's hand-rolled candidate loops could regress subtle existing behavior (e.g., worker's specific WARN logging on standby redirect) — mitigate by preserving equivalent log statements around the new single-call path, verified in code review.
|
||||
- apiserver's circuit breaker could mask a fixed backend as still-degraded if the clearing signal (rejoin callback) doesn't fire in some edge case — mitigate with a bounded TTL on the degraded flag (re-probe after a short timeout) rather than relying solely on the rejoin callback.
|
||||
|
|
|
|||
53
.kiro/specs/component-rejoin-resync/brief.md
Normal file
53
.kiro/specs/component-rejoin-resync/brief.md
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
# Brief: component-rejoin-resync
|
||||
|
||||
## Problem
|
||||
|
||||
When the apiserver or scheduler disconnects and reconnects to the controller (network blip, or reconnect to a new primary after controller failover), each does a full re-join and receives **full** state again (e.g. the whole job list, full queue/resource/config state) rather than a delta. For a large cluster this resend is sizable — bandwidth and re-processing cost scale with cluster size on every reconnect, not just on first join.
|
||||
|
||||
This is a **different problem from `worker-rejoin-resync`**: that spec recovers worker-reported job-*execution facts* (start/finish/fail) that would otherwise be silently lost during a disconnect window. This spec is about the *cost/efficiency* of apiserver/scheduler's full-state resend on reconnect — no data is lost today (full resend is correct), it is potentially just expensive at scale.
|
||||
|
||||
## Current State
|
||||
|
||||
- Full-state rejoin is already accepted, working behavior (`libxce-failover` design review decision): "simple, correct, and still cheaper than a full-history replace with no snapshot" (cf. IBM LSF).
|
||||
- Recorded as a **deferred YAGNI backlog item** ("lightweight rejoin re-attach", 2026-06-16, roadmap.md Future Specs/Backlog) — explicitly *not* promoted previously because no measured problem existed yet.
|
||||
- No delta-resync or session-token re-attach mechanism exists for either component today.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
Not yet decided — this spec starts at the discussion/investigation stage, not a committed design. Candidate outcome: apiserver and/or scheduler reconnect without re-receiving state that is unchanged since their last known position (e.g. keyed on last-acknowledged committed LSN), reducing reconnect cost at scale — IF a real problem is measured. Equally plausible outcome: investigation confirms full-state resend remains fine and this spec closes as "no action needed."
|
||||
|
||||
## Approach
|
||||
|
||||
Not yet chosen. Prior design review flagged the general shape (session-token re-attach hello, or delta-only transfer keyed on last-known committed LSN) but flagged delta reconciliation as "non-trivial (the primary must know what each consumer already holds)" — a real design cost, not a rubber stamp. First step for this spec is likely measurement/investigation (does reconnect cost actually hurt at realistic cluster sizes?) before committing to a mechanism.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: apiserver rejoin content/cost. Scheduler rejoin content/cost. Investigation of whether full-state resend is actually a measured problem before designing a fix.
|
||||
- **Out**: Worker job-execution-fact resync (→ `worker-rejoin-resync`, separate spec, different data shape — facts vs. list-state). Any change to the pending-facts queue. New failover mechanics beyond what `libxce-failover`/`component-failover-adoption` already provide.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- apiserver reconnect path (Go, `src/frontend/apiserver/backend/clib.go` cgo boundary + libxce failover hook).
|
||||
- scheduler reconnect path (`src/scheduler/src/handler.c`, controller-side `on_scheduler_join`/`handle_scheduler_join`, `join.c`).
|
||||
- Measurement/instrumentation to quantify current reconnect cost — likely a prerequisite sub-task before any mechanism design.
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- Worker-side job-fact resync (separate spec, `worker-rejoin-resync`).
|
||||
- Redesigning the join protocol wholesale — this is about reconnect efficiency, not join correctness.
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `libxce-failover` (rejoin hook — implemented). `component-failover-adoption` (daemon rejoin content — currently tasks-generated, in progress) — this spec's scheduler/apiserver reconnect paths overlap with that spec's territory; sequence after or alongside it, not independently duplicating its liveness-safety work.
|
||||
- **Downstream**: none yet.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Adjacent**: `worker-rejoin-resync` — sibling, same trigger (reconnect/failover) but disjoint data (job facts vs. list state). Do not merge scope — different information, different mechanism.
|
||||
- **Adjacent**: `component-failover-adoption` — covers making the daemons *use* libxce failover safely (liveness, master-list config); this spec covers *what* they resync once reconnected, a distinct concern layered on top.
|
||||
- **Supersedes discussion for**: the "lightweight rejoin re-attach" backlog item (roadmap.md) — this spec is that item promoted, scoped to apiserver+scheduler only (worker's share of that item is now `worker-rejoin-resync`).
|
||||
|
||||
## Constraints
|
||||
|
||||
- Not started — investigation/discussion first, no committed mechanism yet.
|
||||
- Should not be pursued as a speculative optimization — prior YAGNI decision stands until this spec's own investigation finds a measured cost, per the same discipline that deferred it originally.
|
||||
24
.kiro/specs/component-rejoin-resync/spec.json
Normal file
24
.kiro/specs/component-rejoin-resync/spec.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"feature_name": "component-rejoin-resync",
|
||||
"created_at": "2026-08-10T00:00:00Z",
|
||||
"updated_at": "2026-08-10T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "brief",
|
||||
"description": "apiserver and scheduler reconnect/rejoin resync efficiency -- full-state resend today; investigate whether a measured cost problem justifies delta/session-token re-attach. Sibling of worker-rejoin-resync (disjoint data: list-state vs job-execution facts).",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": []
|
||||
}
|
||||
|
|
@ -1,9 +1,9 @@
|
|||
{
|
||||
"feature_name": "config-upgrade-snapshot",
|
||||
"created_at": "2026-06-21T11:30:00Z",
|
||||
"updated_at": "2026-07-29T04:10:00Z",
|
||||
"updated_at": "2026-08-15T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
|
@ -19,5 +19,13 @@
|
|||
}
|
||||
},
|
||||
"ready_for_implementation": true,
|
||||
"blocked_on": []
|
||||
"blocked_on": [],
|
||||
"validate_impl": {
|
||||
"decision": "GO",
|
||||
"validated_at": "2026-08-15T00:00:00Z",
|
||||
"note": "Corrected stale phase metadata (was 'tasks-generated' though every task 1-8.9 was already [x] complete). Task 8.5 reproduced bug-857 on the pre-change behavior and confirmed it no longer occurs; task 8.6's live chaotic multi-kill soak (2026-07-29) confirmed zero materialized-state/config divergence after a config upgrade. bug-857 closed as a result.",
|
||||
"known_limitations": [
|
||||
"bug-1471 / bug-1472-remainder: a host admin-closed before the final checkpoint can produce host-frame byte divergence (WAL_WORKER_CTRL apply path, worker.c/persist.c) -- explicitly out of this spec's own materialization boundary per its research.md/tasks.md notes, never converted into a task here. No existing spec currently owns fixing it; flagged for a new task/spec."
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
{
|
||||
"feature_name": "raft-adapter-conformance",
|
||||
"created_at": "2026-07-29T03:10:00Z",
|
||||
"updated_at": "2026-07-29T09:20:00Z",
|
||||
"updated_at": "2026-08-15T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
|
@ -21,5 +21,14 @@
|
|||
"ready_for_implementation": true,
|
||||
"blocked_on": [],
|
||||
"related_bugs": ["bug-1479", "bug-1480", "bug-1481", "bug-1466", "bug-1471", "bug-1472", "bug-1474", "bug-1475", "bug-1477", "bug-1478", "bug-1423"],
|
||||
"related_specs": ["raft-log-replication", "raft-commit-safety", "ha-term-safety", "raft-sim-conformance", "config-upgrade-snapshot", "ha-config-bootstrap", "ha-commit-engine"]
|
||||
"related_specs": ["raft-log-replication", "raft-commit-safety", "ha-term-safety", "raft-sim-conformance", "config-upgrade-snapshot", "ha-config-bootstrap", "ha-commit-engine"],
|
||||
"validate_impl": {
|
||||
"decision": "GO",
|
||||
"validated_at": "2026-08-15T00:00:00Z",
|
||||
"note": "Corrected stale phase metadata (was 'tasks-generated' though every task 1-7 was already [x] complete: config-push reply integrity, vote/pre-vote post-connect failure routing, dormant-sender guard, regression locks, live soaks).",
|
||||
"known_limitations": [
|
||||
"bug-1474 (angle-a only; the repl.c defense-in-depth slot-reuse hardening is already fixed elsewhere): a spurious duplicate/late-joiner reconnect during WAL trim, root-caused to a vote_ep_ctx_t.peer_idx convention mismatch in ha.c's heartbeat-reconnect-sweep (~ha.c:9219-9254). Never converted into a task here despite being tracked in related_bugs -- this spec's tasks 1-7 are entirely about config-push reply integrity and vote/pre-vote failure routing, a different part of ha.c. Natural fit for a new appended task given its scope match.",
|
||||
"bug-1471 / bug-1472-remainder: listed in related_bugs but never converted into a task -- root mechanism (worker.c/persist.c's WAL_WORKER_CTRL apply path) is outside this spec's ha.c/repl.c-adapter boundary; see config-upgrade-snapshot's own known_limitations for the primary tracking."
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
|
|||
58
.kiro/specs/scheduler-sim-diagnostics/brief.md
Normal file
58
.kiro/specs/scheduler-sim-diagnostics/brief.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# Brief: scheduler-sim-diagnostics
|
||||
|
||||
## Problem
|
||||
|
||||
Operators/developers investigating scheduler behavior in simulation mode (the `--sim`/`--interactive` REPL shipped by `scheduler-core`) can inspect job/queue/worker/resource state (`info job/queue/worker/resource/summary`) but cannot answer two common investigation questions:
|
||||
|
||||
1. **What limits are configured, and what is their current usage?** `xce_limit_t`/`xce_vres_t` (limit config + live per-user/queue/project/host usage instances) exist only inside the `limit` plugin's private `kvmgr` tables (`plugin.c`'s `limit_plugin_data_t`) — invisible to the REPL and to `sched_get_*()` accessors.
|
||||
2. **Why did a job pend, or why did a job that was expected to pend actually run?** `xce_pendreason_t` is attached to pend decisions and to `xce_sched_obj_t`, but the REPL never prints it, and for a job that *did* dispatch there is no recorded trail of which limit/resource checks passed — only the final DISPATCH outcome.
|
||||
|
||||
## Current State
|
||||
|
||||
- `scheduler-core` (implementation-complete) shipped the REPL, breakpoints, and state-inspection commands (Req 5 AC 22) — this is the platform this work extends.
|
||||
- No limit accessor exists outside the `limit` plugin's internal tables.
|
||||
- No decision-reasoning trace exists for the dispatch path; only pend decisions carry a reason, and it isn't surfaced.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
Simulation-mode REPL (and, over time, other diagnostic surfaces added under this spec) lets an investigator answer "what is configured / in use" and "why did the scheduler decide X" without reading source or adding printf debugging. This is an **open-ended, continuously-extended spec** — new debug/troubleshoot capabilities get added as requirements over time, driven by discussion or by needs surfacing from other in-flight specs. Not all scope is known upfront.
|
||||
|
||||
## Approach
|
||||
|
||||
Extend the existing REPL/simulation-driver surface (not a new component). Each new capability is its own Requirement added when its scope is settled — this brief captures only the two seed capabilities identified so far; more will be appended.
|
||||
|
||||
**Seed capability 1 — Limit inspection**: expose limit config (name, per_user/queue/project/host scope, value/percent) and live usage (per-vres-instance current value vs limit) via a new `info limit <name|all>` REPL command. Requires plumbing a read path from the `limit` plugin's private kvmgr tables to the REPL (core has no existing accessor for plugin-private state).
|
||||
|
||||
**Seed capability 2 — Decision-reasoning trace**: capture *why* each scheduling decision was made — not just the outcome. For PEND, surface the existing `xce_pendreason_t` (currently captured but not printed). For DISPATCH, there is currently no trail at all of which limit/resource checks were evaluated and passed; this needs new instrumentation in the allocation path (`sched_try_alloc_on_host_`, `limit_prepare_allocation`) to record a per-attempt reasoning trace, then a REPL command to print it for a given job/decision.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: Simulation-mode / REPL-facing investigation tooling only. Read-only inspection — no state mutation (consistent with `scheduler-core` Req 5's existing "no write-side mutations" boundary). Capabilities are added incrementally as separate ACs/Requirements.
|
||||
- **Out**: Live production inspection against a controller-connected scheduler (no `OPCode_schedAdmin`-style wire work) — explicitly deferred; this spec is simulation-mode only per user decision. Any write/mutation commands. New allocation or picker logic — this spec only observes decisions made by existing logic.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Limit inspection (read path + REPL command) — self-contained, touches `limit.c` (expose accessor) + `repl_info.c` (new command).
|
||||
- Decision-reasoning trace (new instrumentation + REPL command) — larger: touches the allocation hot path (`resource.c`, `plugin.c`) to record trace data, plus decision/REPL plumbing to surface it.
|
||||
- Future capabilities: unscoped — to be added as separate requirements when identified.
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- Live/production (controller-connected) diagnostics.
|
||||
- Any REPL command that mutates scheduler state.
|
||||
- New scheduling policy, allocator, or picker behavior.
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `scheduler-core` (REPL, simulation driver, `info` command family this extends). `scheduler-allocation` and `scheduler-preemption` (siblings, currently blocked on scheduler-core, now unblocked) are likely future sources of new diagnostic requirements once they land, since this spec's scope is meant to absorb debug needs "according to the requirement of other projects."
|
||||
- **Downstream**: none yet — open-ended.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `scheduler-core` (REPL/simulation driver architecture, Req 5's `info`/breakpoint/snapshot command family).
|
||||
- **Adjacent**: `scheduler-allocation`, `scheduler-preemption` — not yet implemented; expect this spec to gain requirements once those introduce new decision types/state worth inspecting.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Must not alter deterministic replay guarantees (`scheduler-core` Req 5's normal↔sim equivalence invariant) — inspection commands are read-only and must not perturb core state or event ordering.
|
||||
- Not starting immediately — requirements will be added/refined per-capability over time rather than fully speced upfront.
|
||||
53
.kiro/specs/worker-rejoin-resync/brief.md
Normal file
53
.kiro/specs/worker-rejoin-resync/brief.md
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
# Brief: worker-rejoin-resync
|
||||
|
||||
## Problem
|
||||
|
||||
When a worker disconnects and reconnects (rejoin after network blip, or reconnect to a new primary after controller failover), any job start/finish/fail fact it sent during the disconnected window — and never got acknowledged — is permanently lost. Worker notify today is fire-and-forget with zero redelivery (bug-396). The controller has no way to ask "what happened to my jobs while you were gone," and the worker has no way to answer.
|
||||
|
||||
Evidence: `xce_join_req` (`src/lib/request.c:1398-1429`) only attaches `hostattrs_t` when `role == HostRole_worker` (`joinmsg.data.attrs`) — no job-state payload exists on the join wire message at all. `handle_worker_join`/`handle_join_` (`src/controller/join.c`) process the join purely as a host-attribute registration; there is no code path that reconciles worker-reported job state against the controller's job table on (re)join.
|
||||
|
||||
## Current State
|
||||
|
||||
- `worker-join-refactor` (implemented) covers only attribute/endpoint update on rejoin — no job-state content.
|
||||
- The controller-side pending-facts queue (`ha-consistency-invariants` Req 2.8) covers quorum-loss windows *while the worker stays connected* — it retries facts the controller received but couldn't commit. It does NOT cover facts that never arrived because the worker was disconnected or the controller itself failed over mid-flight. This spec is the designated backstop for that second case (`ha-consistency-invariants` Req 2.8 boundary context, elevated 2026-06-11).
|
||||
- `raft-apply-decoupling` (implemented) elevated this spec to a **hard prerequisite** for the facts half of its own work (2026-06-25) — commit-then-apply idempotency guards this spec needs are now in place.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
On worker rejoin (disconnect/reconnect or failover reconnect), the worker reports its current running + recently-finished job state; the controller reconciles that report against its job table through the existing commit barrier, idempotently (no duplicate facts, no clobbering newer state the controller already has). Facts that would otherwise be silently lost in a disconnect/failover window are recovered.
|
||||
|
||||
## Approach
|
||||
|
||||
Extend the join/hello exchange (or add a post-join follow-up message) so a worker attaches a running+recent-job-state report, analogous to how `hostattrs_t` already rides the join message today. Controller-side, add a reconciliation path that runs this report through `mark_job_*`-style idempotent transitions (reusing the commit-then-apply guards `raft-apply-decoupling`/`ha-consistency-invariants` already built) rather than a new ad hoc mutation path.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: Worker-side job-state rejoin reporting (running jobs + recently-finished/failed jobs not yet acked); controller-side idempotent reconciliation against the job table via the existing commit barrier; wire schema addition to the join exchange (or a new post-join message type); regression coverage for disconnect-during-job and failover-during-job scenarios.
|
||||
- **Out**: apiserver/scheduler resync-on-reconnect (different data — cluster/queue/job *list* state, not job *execution facts* — tracked separately, see roadmap's deferred "lightweight rejoin re-attach" item, YAGNI until measured). Delta/incremental state transfer optimizations (session-token re-attach) — separate, deferred backlog item, not this spec's concern. Any change to the primary-side pending-facts queue itself (already implemented, this spec only complements it).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Wire/schema: extending `xce_join_req`/join message (or new message type) to carry job-state report — worker + controller.capnp touchpoint.
|
||||
- Worker-side: collecting "running + recently-finished-but-unacked" job state at rejoin time (worker.c / controller_connector.c).
|
||||
- Controller-side: reconciliation logic reusing `mark_job_update`/`mark_job_fail`-style idempotent commit-then-apply transitions (job.c).
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- apiserver rejoin content (full state resend — separate, deferred item).
|
||||
- scheduler rejoin content (separate, deferred item).
|
||||
- Delta-based / session-token re-attach optimization (separate backlog item, explicit YAGNI until measured).
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream (hard deps)**: `ha-consistency-invariants` (commit barrier + idempotency guards — implemented). `raft-apply-decoupling` (commit-then-apply seam this reuses — implemented, and the thing that elevated this spec to hard-prereq status for its own facts half).
|
||||
- **Downstream**: `job-array-single-wal-record` and the broader Raft Apply-Ordering family reference this as a complements-not-replaces sibling to the pending-facts queue. The deferred "lightweight rejoin re-attach" backlog item lists this as one of its own dependencies (worker job-state content).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `worker-join-refactor` (implemented — attribute/endpoint update only; this adds the job-state content that was explicitly out of that spec's scope).
|
||||
- **Adjacent**: `ha-consistency-invariants` (Req 2.8 designates this as the official backstop — do not duplicate the pending-facts queue's mechanism, complement it). `component-failover-adoption` (in-progress sibling — makes daemons exercise libxce failover; does not itself cover job-state resync content).
|
||||
|
||||
## Constraints
|
||||
|
||||
- Must not introduce a second, competing idempotency mechanism — reconciliation goes through the same commit-then-apply guards the pending-facts queue and `raft-apply-decoupling` already use.
|
||||
- Must not regress `worker-join-refactor`'s existing attribute/endpoint update behavior on rejoin.
|
||||
24
.kiro/specs/worker-rejoin-resync/spec.json
Normal file
24
.kiro/specs/worker-rejoin-resync/spec.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"feature_name": "worker-rejoin-resync",
|
||||
"created_at": "2026-08-10T00:00:00Z",
|
||||
"updated_at": "2026-08-10T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "brief",
|
||||
"description": "Worker re-reports running/finished job state on rejoin (disconnect or controller failover) so job-execution facts lost during the disconnected window are recovered via commit-then-apply idempotent reconciliation.",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": []
|
||||
}
|
||||
|
|
@ -38,7 +38,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
|
||||
- [x] **raft-drive-async** — **IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks; the live sequential soak this validation ran surfaced a real crash, bug-1447 — a same-batch endpoint use-after-free in `io_worker`'s epoll dispatch loop, exposed by this spec's own faster reconnect cadence, see `ha-async-io.md`; root-caused, fixed, re-soaked clean, re-validated GO same day). Original description retained below for provenance. *(was: START FIRST — threatens failover liveness TODAY)* — bring the code into compliance with the corrected `ha-async-io.md` thread-role contract: dispatch `peer_ctrl_msg_handler` to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse `xce_probe_config_tag` pattern); add a general non-IO vtask-submit path + admission reservation (extend `COMMIT_WAIT_RESERVED`). Verify with a **partition** (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); `election-iomgr-refactor` is completed upstream (provides `delay_task_ms`/`on_connected`/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + **design + tasks done**; phase=tasks-generated (tasks `approved: false`). **`/kiro-validate-design` GO 2026-06-25** — three review findings folded into design+tasks: (1) **deadlock-freedom + LivenessAdmission** — the documented invariant reversal (ack/`commit_on_ack` moves off iomgr to the vtask pool, `commit.c:16`) co-locates commit-barrier wakers+waiters; review found `COMMIT_WAIT_RESERVED` is **insufficient** because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a **pool-level task-class admission in vtaskmgr** (`VTASK_LIVENESS`/`LIVE_BLOCK`/`REQ_BLOCK`; liveness never gated; `COMMIT_WAIT_RESERVED` folded in) — new **Phase-1 task 1.6** (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until `raft-apply-decoupling` P1 makes the barrier async; (2) **WAL-trim cursor-bound guard** added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) **in-process partition primitive** in `xce_test` as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. **Next: approve tasks (`/kiro-spec-tasks raft-drive-async`) → `/kiro-impl`; ship Phase 1 (bounded connect) first.**
|
||||
- [x] raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single `apply_committed_entry()` seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). **Contains P1 (commit-barrier async) as its keystone.** Commands (WAL_MODE_WAIT) apply-after-commit; **facts (WAL_MODE_RETAIN) routed through commit-then-apply too** — D1 RESOLVED (block-but-durable under quorum loss); formalized as `ha-consistency-invariants` criterion 2.9. Dependencies: raft-drive-async, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync; coordinate with ha-term-safety. Briefs: `.kiro/specs/raft-apply-decoupling/brief.md` + `facts-in-raft.md`. **IMPLEMENTED 2026-07-07: 32/32 tasks, validate-impl GO** — 79/79 controller_unit_test groups (926 tests) green, live 3-node smoke boot verified, zero blocked tasks. Task 7.2 fixed bug-1416 (boot-time TOCTOU false APPLY_DIVERGENT fence). Task 7.4's own throughput benchmark found and fixed a real O(n²) apply-pump regression (56% throughput drop, superlinear scaling) via a segment-read cache on `persist_t`, 2 adversarial review rounds. Residual, non-blocking: design.md has 3 sections not yet reconciled with implementation deviations already self-documented in tasks.md's Implementation Notes (repl.c's planned `repl_submit_majority_commit()` was never built; apply pump wired via `commit_resolve_pending()` not `ha.c`'s `on_commit_advanced`; fact hooks collapse to generic `HOOK_DETAIL_JOB_UPDATE` not per-type); task 5.3 touched `ha.c` without a formal `_Boundary:_` exception tag (substantively justified, process-inconsistent only). Two out-of-scope findings flagged for follow-up: an intermittent client hang in the synchronous `jobsub` path (resembles but not confirmed as bug-1416's class), and `trim_wal_segments`'s retention floor has no awareness of the local apply pump's own lag (same family as an already-tracked standby-cursor gap).
|
||||
- [ ] worker-rejoin-resync — **elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25** (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only `hostattrs` on join (bug-396). Already on the backlog below; pulled forward into this family. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards).
|
||||
- [ ] worker-rejoin-resync — **PROMOTED to spec 2026-08-10** (`/kiro-discovery`; brief at `.kiro/specs/worker-rejoin-resync/brief.md`). **Elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25** (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only `hostattrs` on join (bug-396). Scoped narrowly to **worker** job-execution-fact resync only — apiserver/scheduler resync-on-reconnect is a different data shape (cluster/queue/job *list* state, not job *execution facts*) and stays out of scope (see the deferred "lightweight rejoin re-attach" backlog item below, still YAGNI). Dependencies: ha-consistency-invariants (commit barrier + idempotency guards) — both implemented, unblocked. Next: `/kiro-spec-requirements worker-rejoin-resync`.
|
||||
- [ ] job-array-single-wal-record — **PROMOTED from backlog 2026-06-24.** Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: `.kiro/specs/job-array-single-wal-record/brief.md`.
|
||||
|
||||
### Existing Spec Updates (Raft Apply-Ordering phase)
|
||||
|
|
@ -64,9 +64,10 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
|
||||
- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.**
|
||||
- [ ] component-failover-adoption — **recorded 2026-06-16; depends on libxce-failover (HARD).** Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link `libxce_dep`; apiserver uses it via cgo, `src/frontend/apiserver/backend/clib.go`), but two per-component integration items remain: (1) each component must hand libxce the **full master list** instead of a single endpoint (today even the CLI uses only `conf->masters[0]`, `client.c:122`); (2) bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons (worker `controller_connector.c`, scheduler `handler.c`) this must be verified **liveness-safe**, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. (`xce_test` is NOT here — it is the in-spec verification client of libxce-failover.) Promote via `/kiro-spec-init component-failover-adoption` after libxce-failover lands.
|
||||
- [ ] worker-rejoin-resync — **planned long ago, no effort yet (recorded 2026-06-10).** When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): `xce_join_req` carries only `hostattrs_t` (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the `mark_job_*` idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. **Elevated 2026-06-11**: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
|
||||
- [x] ~~worker-rejoin-resync~~ — **PROMOTED to spec 2026-08-10, see In Flight (Raft Apply-Ordering family) above** for current status and brief. No longer tracked here.
|
||||
- [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) was blocked on `entity-bearing-snapshots`; that blocker is now cleared (see Future Specs/Backlog above — `ha-entity-snapshots` implemented, `config-upgrade-snapshot` `blocked_on: []`). Next: `/kiro-spec-requirements admin-config-tool`.
|
||||
- [ ] lightweight rejoin re-attach (failover optimization) — **recorded 2026-06-16; deferred (YAGNI).** Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state).
|
||||
- [ ] **component-rejoin-resync** — **PROMOTED to spec 2026-08-10** (`/kiro-discovery`; brief at `.kiro/specs/component-rejoin-resync/brief.md`) from the "lightweight rejoin re-attach" item below. **apiserver + scheduler** reconnect/rejoin resync efficiency — today each does a full re-join and receives full state again (whole job list, queue/resource/config state), not a delta; for a large cluster this is sizable but not yet a measured problem. Sibling of `worker-rejoin-resync` — disjoint data (list/config state vs. job-execution facts), NOT merged into that spec. Not started: first step is investigation (is reconnect cost actually a measured problem at realistic cluster sizes?) before committing to any delta/session-token mechanism — original YAGNI discipline still applies, this spec is the promoted vehicle for that investigation, not a rubber-stamped design. Original decision (libxce-failover design review) retained below for provenance. Dependencies: libxce-failover (rejoin hook — implemented), component-failover-adoption (daemon rejoin content — in progress, sequence alongside/after). Next: `/kiro-spec-requirements component-rejoin-resync` when ready to begin (not immediate).
|
||||
- [x] ~~lightweight rejoin re-attach (failover optimization)~~ — **recorded 2026-06-16; deferred (YAGNI)**; apiserver/scheduler half **PROMOTED to `component-rejoin-resync` 2026-08-10** (see above); worker half already promoted to `worker-rejoin-resync`. No longer tracked as a bare backlog item — provenance only. Original text: on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale.
|
||||
- [x] job-array-single-wal-record — **IMPLEMENTED, validate-impl GO 2026-07-17** (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
|
||||
- [ ] bulk job-array query/cancel — **recorded 2026-07-13, deferred out of `job-array-single-wal-record` scope by design.** A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for `job-array-single-wal-record` — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array *submission* and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: `job-array-single-wal-record` (per-element identity/addressing this would operate over). Promote via `/kiro-discovery` once job-array-single-wal-record ships.
|
||||
- [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
|
||||
|
|
@ -92,6 +93,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
|
|||
- [x] scheduler framework enhancement — **PROMOTED to spec 2026-07-06, then DECOMPOSED 2026-07-27** into `scheduler-core` (**IMPLEMENTATION-COMPLETE**), `scheduler-allocation`, and `scheduler-preemption` — see **In Flight** above for current status. Scope notes from the original brief still apply: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). No longer tracked here.
|
||||
- [→] **perf-tuning-extraction** — **PROMOTED to spec 2026-07-20** (`clusterconfig-schema-split`, `.kiro/specs/clusterconfig-schema-split/`). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — `candidates`/`vcpus`/hash-tuning fields (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`) are already dead weight in the `ClusterConfig` capnp blob (superseded by `--peers`/`--vcpus` CLI flags or never read into runtime state — see `src/docs/cluster-config-split-analysis.md`). **`port` is NOT dead** — a second review pass (2026-07-20) found `cluster_open_listener` (`cluster.c:1216-1258`) falls back to the blob's `port` field, and single-node mode (no `--peers` flag at all, `main.c:293-294`) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (`clusterconfig-schema-split` Requirement 1.7), not a bare deletion. `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN `ClusterConfig` for operational uniformity despite passing the independence rule — see `.kiro/steering/config-design.md`. `fetchJobInterval`/`netioTimeout` left unresolved, flagged out-of-scope in the new spec for a future pass.
|
||||
- [ ] **Raft/HA heartbeat interval+timeout as CLI parameters** — **NEW 2026-07-20 from clusterconfig-schema-split discovery.** The controller-to-controller HA/Raft election heartbeat (`ha->config.heartbeat_interval_ms`/`heartbeat_timeout_ms` — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (`HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS`, `cluster.c:1131`) — there is no `ClusterConfig` field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: `--heartbeat-interval-ms`/`--heartbeat-timeout-ms` CLI flags following the exact `--vcpus` pattern (`OPT_*` + `struct arguments` + `myconf_t` field, `0` sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking `clusterconfig-schema-split`.
|
||||
- [ ] **scheduler-sim-diagnostics** — **NEW 2026-08-10 (`/kiro-discovery`).** Open-ended, continuously-extended spec for scheduler simulation-mode debug/troubleshoot capability — added to over time as new investigation needs surface from discussion or from other in-flight scheduler specs (`scheduler-allocation`, `scheduler-preemption`). Extends `scheduler-core`'s REPL (`info job/queue/worker/resource/summary`, breakpoints, snapshots). Two seed capabilities identified so far: (1) **limit inspection** — `xce_limit_t`/`xce_vres_t` config + live usage currently live only in the `limit` plugin's private kvmgr tables, invisible to the REPL; needs a new `info limit <name|all>` command + accessor plumbing. (2) **decision-reasoning trace** — `xce_pendreason_t` is captured on pend decisions but never printed by the REPL, and dispatch decisions have no reasoning trail at all (no record of which limit/resource checks passed) — needs new instrumentation in the allocation path (`resource.c`'s `sched_try_alloc_on_host_`, `plugin.c`'s `limit_prepare_allocation`) plus a REPL command to surface it. Explicitly simulation-mode only (no live/production controller-connected diagnostics) and read-only (no state mutation) per user decision 2026-08-10. Brief: `.kiro/specs/scheduler-sim-diagnostics/brief.md`. Not started — requirements to be added per-capability, not fully speced upfront. Next: `/kiro-spec-init scheduler-sim-diagnostics` when ready to begin.
|
||||
|
||||
## Superseded / Absorbed Items
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue