docs(kiro): reconcile spec metadata, promote 3 new specs, fix template path
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
3f58a3636b
commit
ce0ee78b0f
18 changed files with 467 additions and 114 deletions
53
.kiro/specs/worker-rejoin-resync/brief.md
Normal file
53
.kiro/specs/worker-rejoin-resync/brief.md
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
# Brief: worker-rejoin-resync
|
||||
|
||||
## Problem
|
||||
|
||||
When a worker disconnects and reconnects (rejoin after network blip, or reconnect to a new primary after controller failover), any job start/finish/fail fact it sent during the disconnected window — and never got acknowledged — is permanently lost. Worker notify today is fire-and-forget with zero redelivery (bug-396). The controller has no way to ask "what happened to my jobs while you were gone," and the worker has no way to answer.
|
||||
|
||||
Evidence: `xce_join_req` (`src/lib/request.c:1398-1429`) only attaches `hostattrs_t` when `role == HostRole_worker` (`joinmsg.data.attrs`) — no job-state payload exists on the join wire message at all. `handle_worker_join`/`handle_join_` (`src/controller/join.c`) process the join purely as a host-attribute registration; there is no code path that reconciles worker-reported job state against the controller's job table on (re)join.
|
||||
|
||||
## Current State
|
||||
|
||||
- `worker-join-refactor` (implemented) covers only attribute/endpoint update on rejoin — no job-state content.
|
||||
- The controller-side pending-facts queue (`ha-consistency-invariants` Req 2.8) covers quorum-loss windows *while the worker stays connected* — it retries facts the controller received but couldn't commit. It does NOT cover facts that never arrived because the worker was disconnected or the controller itself failed over mid-flight. This spec is the designated backstop for that second case (`ha-consistency-invariants` Req 2.8 boundary context, elevated 2026-06-11).
|
||||
- `raft-apply-decoupling` (implemented) elevated this spec to a **hard prerequisite** for the facts half of its own work (2026-06-25) — commit-then-apply idempotency guards this spec needs are now in place.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
On worker rejoin (disconnect/reconnect or failover reconnect), the worker reports its current running + recently-finished job state; the controller reconciles that report against its job table through the existing commit barrier, idempotently (no duplicate facts, no clobbering newer state the controller already has). Facts that would otherwise be silently lost in a disconnect/failover window are recovered.
|
||||
|
||||
## Approach
|
||||
|
||||
Extend the join/hello exchange (or add a post-join follow-up message) so a worker attaches a running+recent-job-state report, analogous to how `hostattrs_t` already rides the join message today. Controller-side, add a reconciliation path that runs this report through `mark_job_*`-style idempotent transitions (reusing the commit-then-apply guards `raft-apply-decoupling`/`ha-consistency-invariants` already built) rather than a new ad hoc mutation path.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: Worker-side job-state rejoin reporting (running jobs + recently-finished/failed jobs not yet acked); controller-side idempotent reconciliation against the job table via the existing commit barrier; wire schema addition to the join exchange (or a new post-join message type); regression coverage for disconnect-during-job and failover-during-job scenarios.
|
||||
- **Out**: apiserver/scheduler resync-on-reconnect (different data — cluster/queue/job *list* state, not job *execution facts* — tracked separately, see roadmap's deferred "lightweight rejoin re-attach" item, YAGNI until measured). Delta/incremental state transfer optimizations (session-token re-attach) — separate, deferred backlog item, not this spec's concern. Any change to the primary-side pending-facts queue itself (already implemented, this spec only complements it).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Wire/schema: extending `xce_join_req`/join message (or new message type) to carry job-state report — worker + controller.capnp touchpoint.
|
||||
- Worker-side: collecting "running + recently-finished-but-unacked" job state at rejoin time (worker.c / controller_connector.c).
|
||||
- Controller-side: reconciliation logic reusing `mark_job_update`/`mark_job_fail`-style idempotent commit-then-apply transitions (job.c).
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- apiserver rejoin content (full state resend — separate, deferred item).
|
||||
- scheduler rejoin content (separate, deferred item).
|
||||
- Delta-based / session-token re-attach optimization (separate backlog item, explicit YAGNI until measured).
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream (hard deps)**: `ha-consistency-invariants` (commit barrier + idempotency guards — implemented). `raft-apply-decoupling` (commit-then-apply seam this reuses — implemented, and the thing that elevated this spec to hard-prereq status for its own facts half).
|
||||
- **Downstream**: `job-array-single-wal-record` and the broader Raft Apply-Ordering family reference this as a complements-not-replaces sibling to the pending-facts queue. The deferred "lightweight rejoin re-attach" backlog item lists this as one of its own dependencies (worker job-state content).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `worker-join-refactor` (implemented — attribute/endpoint update only; this adds the job-state content that was explicitly out of that spec's scope).
|
||||
- **Adjacent**: `ha-consistency-invariants` (Req 2.8 designates this as the official backstop — do not duplicate the pending-facts queue's mechanism, complement it). `component-failover-adoption` (in-progress sibling — makes daemons exercise libxce failover; does not itself cover job-state resync content).
|
||||
|
||||
## Constraints
|
||||
|
||||
- Must not introduce a second, competing idempotency mechanism — reconciliation goes through the same commit-then-apply guards the pending-facts queue and `raft-apply-decoupling` already use.
|
||||
- Must not regress `worker-join-refactor`'s existing attribute/endpoint update behavior on rejoin.
|
||||
24
.kiro/specs/worker-rejoin-resync/spec.json
Normal file
24
.kiro/specs/worker-rejoin-resync/spec.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"feature_name": "worker-rejoin-resync",
|
||||
"created_at": "2026-08-10T00:00:00Z",
|
||||
"updated_at": "2026-08-10T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "brief",
|
||||
"description": "Worker re-reports running/finished job state on rejoin (disconnect or controller failover) so job-execution facts lost during the disconnected window are recovered via commit-then-apply idempotent reconciliation.",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": []
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue