forked from lei.chen/myxce
save wolf state and document
This commit is contained in:
parent
c68935127c
commit
ea0ac81c92
9 changed files with 30215 additions and 429 deletions
586
.kiro/specs/admin-config-tool/brief.md
Normal file
586
.kiro/specs/admin-config-tool/brief.md
Normal file
|
|
@ -0,0 +1,586 @@
|
|||
# Brief: admin-config-tool
|
||||
|
||||
## Problem
|
||||
|
||||
XCE operators have no tool to build and deploy cluster configuration. Currently:
|
||||
|
||||
- **Config authoring**: YAML config files (`configs/hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`) exist as documentation examples with JSON schemas for validation (`configs/schema/*.schema.json`), but **no tool compiles them** into the `ClusterConfig` Cap'n Proto binary that the controller expects.
|
||||
- **Config deployment**: `OP_upgradeConfig` exists on the controller side (`config_repl.c:1536` — receives binary blob, verifies SHA-256, saves, pushes to standbys, awaits majority ACK, signals restart), but **no CLI sends it**. Today: manual scp + restart.
|
||||
|
||||
End users use `xcecli` (Go/Cobra → REST → apiserver → CGO → controller) for non-admin operations. There is no admin-facing tool.
|
||||
|
||||
**Who**: XCE cluster operators. **Pain**: manual config management, error-prone YAML→binary translation, no primary auto-discovery.
|
||||
|
||||
## Current State
|
||||
|
||||
| What | Where | Status |
|
||||
|------|-------|--------|
|
||||
| `ClusterConfig` capnp struct | `src/schema/controller.capnp:710-732` | 21 fields, wire-ready |
|
||||
| YAML config examples | `configs/{hosts,queues,resources,users}.yaml` | Human-authored, NOT loaded by any code |
|
||||
| JSON schemas | `configs/schema/*.schema.json` | JSON Schema 2020-12 |
|
||||
| `OP_upgradeConfig` handler | `config_repl.c:1536` `do_upgrade_config_with_bytes()` | Receives binary, verifies, pushes, awaits majority |
|
||||
| `UpgradeConfigRequest/Reply` | `controller.capnp:792-797` | Defined in schema |
|
||||
| libxce upgrade API | `request.c:1046` has extractors only | No send function |
|
||||
| Controller wire protocol | TCP+TLS, capnp-encoded `Request`/`Reply`, same CA as controller-to-controller | Exists, used by all C daemons |
|
||||
|
||||
**Key gap**: YAML → ClusterConfig binary tool does not exist. Upload CLI does not exist.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
Single `xceadmin` Python CLI that operators use for cluster configuration management:
|
||||
|
||||
```
|
||||
# Build: auto-discover YAML files + schemas from standard directory layout
|
||||
xceadmin config build configs/ # default output: cluster.bin
|
||||
xceadmin config build configs/ -o cluster.bin # explicit output
|
||||
xceadmin config build configs/ -o - # stdout (for piping)
|
||||
|
||||
# Push: zero-config dev (same as xce_test), single --controller for production
|
||||
xceadmin config push cluster.bin # default: localhost:9991
|
||||
xceadmin config push --controller ctrl-1:9991 cluster.bin # production: any one controller
|
||||
```
|
||||
|
||||
**Standard config directory layout** (auto-discovered):
|
||||
```
|
||||
configs/
|
||||
├── hosts.yaml # optional — missing = empty hosts section
|
||||
├── queues.yaml # optional
|
||||
├── resources.yaml # optional
|
||||
├── users.yaml # optional
|
||||
└── schema/ # required for validation
|
||||
├── hosts.schema.json
|
||||
├── queues.schema.json
|
||||
├── resources.schema.json
|
||||
└── users.schema.json
|
||||
```
|
||||
|
||||
- **config build**: Scan directory → validate present YAML files against corresponding JSON schemas → map to `ClusterConfig` capnp struct → serialize to `cluster.bin` → output SHA-256. Missing files = empty section (not error). Schema directory must exist.
|
||||
- **config push**: Read `cluster.bin` → connect to controller (default `localhost:9991`) → probe current `configGeneration` → if binary's generation ≤ current, increment to `current + 1` and re-encode (controller rejects non-increasing generations as no-op) → controller reply includes peer list + primary address → follow redirect if needed → send capnp `Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}}` → receive `Reply` → display `{versionTag, ackedCount, error}`. Admin can force a specific generation with `--generation N`.
|
||||
- **Shared package**: `xcecommon` Python package with capnp schema handling, wire protocol, and config models — reusable by future Python-based tools
|
||||
|
||||
## Approach
|
||||
|
||||
**Chosen: Python CLI with single dependency on `controller.capnp`.**
|
||||
|
||||
```
|
||||
src/schema/controller.capnp ← single source of truth
|
||||
│
|
||||
├──▶ c-capnproto → libxce → controller (C daemon, existing)
|
||||
│
|
||||
└──▶ pycapnp → xceadmin (Python, new)
|
||||
│
|
||||
├── config build: YAML → ClusterConfig → bytes → cluster.bin
|
||||
└── config push: Request{OP_upgradeConfig, ...} → TLS → Reply
|
||||
```
|
||||
|
||||
- **Language**: Python. YAML parsing trivial (`yaml`), JSON Schema validation (`jsonschema`), capnp encoding (`pycapnp`), CLI (`click` or `argparse`).
|
||||
- **Protocol**: TCP+TLS directly to controller (NOT through apiserver). Zero-config dev experience — default `localhost:9991`, same as xce_test. Single `--controller` flag for production override.
|
||||
- **Dependencies**: `controller.capnp` schema file only. Zero CGO, zero libxce. Pure Python, open-source ready.
|
||||
- **Auto primary detection**: Connect to given controller → controller reply carries peer list + primary address (standby redirect, already in reply schema) → follow redirect to primary → send request. One controller is enough to bootstrap full cluster discovery. Same mechanism xce_test uses
|
||||
- **No CGO**: Wire protocol is TLS connect → send one capnp `Request` → receive one capnp `Reply` → exit. ~150 lines of Python. No iomgr event loop, no vtask thread pool, no persistent connections — those are for daemons, not one-shot CLI tools.
|
||||
|
||||
### Architecture Decisions (from 2026-07-06 discovery session)
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| **Python** over Go | Future open-source release; shared capnp infra with planned Python xcecli rewrite; `pip install` no compilation |
|
||||
| **Pure Python wire** over CGO+libxce | Open-source friendly; no C compilation; wire protocol is one-shot send/receive (~150 lines) |
|
||||
| **Direct controller** over apiserver | Admin path needs no RBAC middleware; apiserver may be restarting during config upgrade; controller SSL is same CA |
|
||||
| **Single `controller.capnp` dependency** | Schema IS the protocol contract; versioned; no extra API surface |
|
||||
| **Separate `xceadmin` binary** over extending xcecli | `xcecli` = user tool (HTTP → apiserver); `xceadmin` = operator tool (TLS → controller); different audiences, different protocols |
|
||||
| **Auto primary detection** | Operator shouldn't need to know which controller is primary; connect to any one, discover peers + primary from reply |
|
||||
| **Config build offline** | YAML→binary is pure data transform; no controller connection needed; cluster.bin can be pre-built, reviewed, version-controlled |
|
||||
|
||||
### xcecli vs xceadmin
|
||||
|
||||
| | xcecli | xceadmin |
|
||||
|---|--------|----------|
|
||||
| **Audience** | End users | Operators |
|
||||
| **Protocol** | HTTP/HTTPS (REST, JSON) | TCP+TLS (capnp binary) |
|
||||
| **Target** | apiserver | controller (primary) |
|
||||
| **Auth** | JWT token (login → apiserver) | None — same zero-config TLS as xce_test |
|
||||
| **Auth middleware** | Full: WhitelistURL→WhitelistIP→RateLimit→JWT→CORS→RBAC | None — direct TCP+TLS, same trust model as controller peers |
|
||||
| **Use cases** | Submit jobs, list queues, check status | Push config, cluster admin, force snapshot |
|
||||
| **Language** | Go/Cobra (current), Python (future) | Python |
|
||||
|
||||
## Project Decomposition (from 2026-07-06 discovery)
|
||||
|
||||
This discovery session (Path E — mixed decomposition) identified 3 workstreams:
|
||||
|
||||
| # | Workstream | Type | Language | Dependencies |
|
||||
|---|-----------|------|----------|--------------|
|
||||
| 1 | **admin-config-tool** | NEW spec | Python | `controller.capnp` only |
|
||||
| 2 | **config-upgrade-snapshot** | EXISTING spec extension | C | entity-bearing-snapshots (blocked upstream) |
|
||||
| 3 | **perf-tuning-extraction** | DIRECT work | doc | None |
|
||||
|
||||
**Dependency flow:**
|
||||
```
|
||||
admin-config-tool (standalone, ships first)
|
||||
│
|
||||
│ config push triggers controller-side path
|
||||
▼
|
||||
config-upgrade-snapshot (controller: replicate → snapshot → restart → activate)
|
||||
│
|
||||
│ blocked on
|
||||
▼
|
||||
entity-bearing-snapshots (upstream blocker, tracked in roadmap)
|
||||
```
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**:
|
||||
- `xceadmin config build`: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. `configGeneration` defaults to 0 (offline-safe).
|
||||
- `xceadmin config push`: single controller address (default `localhost:9991`), probe current configGeneration from controller, auto-increment if needed, auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. `--generation N` flag for explicit override.
|
||||
- `xceadmin config diff`: (nice-to-have) decode two cluster.bin files, human-readable field-by-field diff. Pre-push safety: "what will this config change actually do?"
|
||||
- `xceadmin config push --dry-run`: (nice-to-have) validate controller will accept the config without committing it
|
||||
- `xcecommon` shared Python package: capnp schema loading, wire protocol (TLS+capnp framing), config model types
|
||||
- Build system integration: pip-based install (editable for dev, pinned for release), nix dev shell packages, xce-build.sh component target
|
||||
- Skills integration (see Skill Integration section below):
|
||||
- `xce-build`: add `xceadmin` component target (Python package build + test)
|
||||
- `xce-deploy-ha`: replace manual `ha/data/cluster.bin` prerequisite with `xceadmin config build` step; add `--build-config` flag for on-the-fly YAML→binary generation
|
||||
- `xce-test-ha`: (future) config push validation round in soak loop — deferred until `config-upgrade-snapshot` lands on controller side
|
||||
- Validation: reject configs using reserved name `"lost_and_found"` for queue/host (defense-in-depth — controller enforces at runtime via FATAL, tool prevents at authoring time)
|
||||
- **Out**:
|
||||
- Controller-side config replication/snapshot/restart logic (owned by `config-upgrade-snapshot`)
|
||||
- Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`, upstream blocker)
|
||||
- Performance tuning parameter extraction (separate direct work item)
|
||||
- xcecli Python rewrite (future, separate spec)
|
||||
- xce-test-ha config push validation round (deferred until `config-upgrade-snapshot` lands — config push to a controller that can't replicate/snapshot is a no-op from the soak's perspective)
|
||||
|
||||
## Skill Integration
|
||||
|
||||
Three existing skills touch `cluster.bin` — the binary config blob that `xceadmin config build` produces. Currently `cluster.bin` is a hand-crafted opaque artifact with no visible provenance. `xceadmin` replaces that with a reproducible build step.
|
||||
|
||||
### xce-build — add `xceadmin` component
|
||||
|
||||
New component target for the Python package:
|
||||
|
||||
```
|
||||
/xce-build xceadmin # pip install -e . (editable, for dev)
|
||||
/xce-build xceadmin release # pip install . (pinned, for production)
|
||||
```
|
||||
|
||||
Build path differs from C components: Python, not C. Runs inside nix dev shell (pycapnp needs capnproto C++ runtime from nix). Build output: `xceadmin` CLI entry point + `xcecommon` package importable.
|
||||
|
||||
This is the **blocking** integration — other skills can't use xceadmin until this lands.
|
||||
|
||||
### xce-deploy-ha — generate cluster.bin from YAML
|
||||
|
||||
**Current flow:**
|
||||
```
|
||||
ha/data/cluster.bin (pre-built, manual, opaque) → deploy-ha.sh → controller/ctrl-1/conf/
|
||||
```
|
||||
|
||||
**New flow:**
|
||||
```
|
||||
configs/*.yaml → xceadmin config build → cluster.bin → deploy-ha.sh → ...
|
||||
```
|
||||
|
||||
Changes to the skill:
|
||||
|
||||
| What | Before | After |
|
||||
|------|--------|-------|
|
||||
| Prerequisites | `cluster.bin` must exist under `ha/data/` | `cluster.bin` built from `configs/*.yaml` via `xceadmin config build` |
|
||||
| Config build step | None (manual) | `xceadmin config build configs/ -o ha/data/cluster.bin` (auto, or skip if `--use-existing-config`) |
|
||||
| `--build-config` flag | N/A | New flag: force rebuild cluster.bin from YAML before deploy |
|
||||
|
||||
`deploy-ha.sh` gains an optional precondition: if `cluster.bin` missing or `--build-config` passed, run `xceadmin config build`. Otherwise use existing `cluster.bin` as-is (backward compatible — operators can still hand-craft for custom configs).
|
||||
|
||||
### xce-test-ha — config push validation (future)
|
||||
|
||||
Deferred until `config-upgrade-snapshot` lands on controller side. When ready:
|
||||
|
||||
- **New soak round type**: kill primary, push new config via `xceadmin config push`, verify controller accepted + standbys received
|
||||
- **Coverage**: exercises `OP_upgradeConfig` → replication → majority ACK path under failover
|
||||
- **Verdict**: config push failure = FAIL (config consistency is a data-safety property)
|
||||
|
||||
Not scoped into admin-config-tool tasks — belongs to `config-upgrade-snapshot` spec. Listed here for cross-spec visibility.
|
||||
|
||||
### Integration dependency order
|
||||
|
||||
```
|
||||
xce-build (+xceadmin) ← THIS SPEC, blocking
|
||||
│
|
||||
▼
|
||||
xce-deploy-ha ← THIS SPEC, immediate after xce-build
|
||||
│
|
||||
▼
|
||||
xce-test-ha ← config-upgrade-snapshot spec (future)
|
||||
(config push round) blocked on entity-bearing-snapshots
|
||||
```
|
||||
|
||||
## Harness Enhancement
|
||||
|
||||
The `xceadmin` tool is Python — first Python component in a C/Go project. This requires harness changes across the development toolchain.
|
||||
|
||||
### xce-build.sh — new `xceadmin` component
|
||||
|
||||
Add a new case to `src/tools/xce-build.sh`:
|
||||
|
||||
```bash
|
||||
xceadmin)
|
||||
echo "=== Building xceadmin Python package ==="
|
||||
pip install -e . --quiet
|
||||
echo "=== Running xceadmin tests ==="
|
||||
python -m pytest tests/ -v
|
||||
;;
|
||||
```
|
||||
|
||||
Python path — no meson setup. `pip install -e .` for dev (editable), `pip install .` for release. Tests via `pytest`, not meson test. Runs inside nix dev shell (capnproto C++ runtime needed for pycapnp compile). The `release` profile uses `pip install .` without `-e`.
|
||||
|
||||
### xce-build SKILL.md — add component + usage note
|
||||
|
||||
- **Usage line**: `Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin`
|
||||
- **Profile notes**: `xceadmin` ignores meson profiles — it always uses pip. The profile arg affects C components only; xceadmin builds regardless of profile.
|
||||
- **Examples**: `/xce-build xceadmin` (editable), `/xce-build xceadmin release` (pinned)
|
||||
|
||||
### xce-deploy-ha SKILL.md — config build step
|
||||
|
||||
Changes to the skill document:
|
||||
|
||||
- **Prerequisites**: replace "TLS certs and a `cluster.bin` must exist under `ha/data/`" with "TLS certs must exist under `ha/data/`. `cluster.bin` is built from `configs/*.yaml` via `xceadmin config build` (or use existing if `--use-existing-config`)."
|
||||
- **New `--build-config` flag**: forces rebuild of `cluster.bin` from YAML before deploy. Without it, uses existing `cluster.bin` as-is (backward compatible).
|
||||
- **New step** (between resolve paths and stop running): optional `xceadmin config build configs/ -o ha/data/cluster.bin` if `--build-config` passed or `cluster.bin` missing.
|
||||
- **Troubleshooting**: add "`pycapnp` import error" → run `/xce-build xceadmin` first.
|
||||
|
||||
### settings.local.json — Python permissions
|
||||
|
||||
Add Bash permission allowlist entries for xceadmin development:
|
||||
|
||||
```json
|
||||
"Bash(pip install *)",
|
||||
"Bash(python -m pytest *)",
|
||||
"Bash(xceadmin config build *)",
|
||||
"Bash(xceadmin config push *)",
|
||||
"Bash(ruff format *)",
|
||||
"Bash(ruff check *)"
|
||||
```
|
||||
|
||||
### PostToolUse hooks — Python formatting + lint
|
||||
|
||||
Add a ruff format hook for Python file edits (`settings.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"matcher": "Write|Edit",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "[[ \"$FILE_PATH\" == *.py ]] && ruff format --quiet \"$FILE_PATH\" || true",
|
||||
"timeout": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Add a ruff check hook for lint feedback:
|
||||
|
||||
```json
|
||||
{
|
||||
"matcher": "Write|Edit",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "[[ \"$FILE_PATH\" == *.py ]] && ruff check --quiet \"$FILE_PATH\" || true",
|
||||
"timeout": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### PreToolUse hook — YAML schema guard (nice-to-have)
|
||||
|
||||
Validate config YAML files against JSON schemas before accepting edits. Prevents commits of invalid config:
|
||||
|
||||
```json
|
||||
{
|
||||
"matcher": "Write|Edit",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "[[ \"$FILE_PATH\" == configs/*.yaml ]] && xceadmin config validate \"$FILE_PATH\" || true",
|
||||
"timeout": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This requires `xceadmin config validate` subcommand (nice-to-have: validates a single YAML file against its schema, no binary output). Deferred to a follow-up task after core `config build` lands.
|
||||
|
||||
### `/xce-admin` skill (deferred)
|
||||
|
||||
Thin skill wrapper around `xceadmin config build/push` for agent-driven workflows. Not needed for spec implementation — CLI is sufficient. Deferred until CLI is stable and operator workflows are validated.
|
||||
|
||||
### Harness change summary
|
||||
|
||||
| # | What | Where | Priority | Phase |
|
||||
|---|------|-------|----------|-------|
|
||||
| 1 | `xceadmin` case in build script | `src/tools/xce-build.sh` | **Blocking** | Implementation |
|
||||
| 2 | Component list + pip note | `.claude/skills/xce-build/SKILL.md` | **Blocking** | Implementation |
|
||||
| 3 | `--build-config` + config build step | `.claude/skills/xce-deploy-ha/SKILL.md` | **Blocking** | Implementation |
|
||||
| 4 | pip/pytest/xceadmin permissions | `.claude/settings.local.json` | **Blocking** | Implementation |
|
||||
| 5 | ruff format hook on `.py` edits | `.claude/settings.json` | Nice-to-have | Implementation |
|
||||
| 6 | YAML schema guard on config edits | `.claude/settings.json` | Nice-to-have | Follow-up |
|
||||
| 7 | `/xce-admin` skill wrapper | `.claude/skills/xce-admin/SKILL.md` | Deferred | Post-spec |
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- **`xceadmin` CLI** — operator-facing commands, argument parsing, output formatting
|
||||
- **`xcecommon` Python package** — shared capnp schema loading, wire protocol (TLS+capnp), config model types. Reusable by future Python xcecli.
|
||||
- **`config-build` module** — YAML→ClusterConfig mapping, JSON Schema validation, serialization
|
||||
- **`config-push` module** — primary auto-detection, TLS+capnp send/receive, result parsing
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `controller.capnp` schema (source of truth for ClusterConfig, Request/Reply, OP codes)
|
||||
- **Downstream**: Controller `OP_upgradeConfig` handler (`config_repl.c` — already exists)
|
||||
- **Consumed by**:
|
||||
- `config-upgrade-snapshot` spec — xceadmin is the trigger; controller-side handles replication/snapshot/restart
|
||||
- `xce-build` skill — adds `xceadmin` component target (blocking integration)
|
||||
- `xce-deploy-ha` skill — `xceadmin config build` replaces manual `cluster.bin` prerequisite
|
||||
- `xce-test-ha` skill — (future) config push validation round, deferred until config-upgrade-snapshot lands
|
||||
- **Future consumer**: Python xcecli rewrite (imports `xcecommon` for capnp schema types)
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Python ≥ 3.10** (match system Python, avoid bleeding-edge features)
|
||||
- **pycapnp** requires capnproto C++ runtime (`libcapnp-dev`) — separate from project's c-capnproto (C bindings). Both capnp runtimes can coexist; binary format is standardized.
|
||||
- **Zero-config TLS** — same dev experience as xce_test: default `localhost:9991`, no cert flags needed. Controller peer trust model (X.509 CN validation via `validate_host()` — tracked in roadmap hardening).
|
||||
- **No dependency on a running apiserver** — admin tool connects directly to controller. Config deployment works even when apiserver is down or restarting.
|
||||
- **Open-source readiness** — pure Python, `pip install` friendly, no compilation step, clear single-schema dependency.
|
||||
|
||||
## Nix Environment
|
||||
|
||||
`xceadmin` is the first Python component in the XCE project. The nix dev shell currently provides C/C++ tooling only — zero Python packages. All needed Python packages are available in nixpkgs.
|
||||
|
||||
### Packages to add
|
||||
|
||||
Current `src/nixSupport/xce.nix` `nativeBuildInputs` already includes `capnproto` (line 21) — the C++ runtime that pycapnp wraps. Need to add:
|
||||
|
||||
| Package | Nix attr | Purpose | Size |
|
||||
|---------|----------|---------|------|
|
||||
| Python 3 | `python3` | Interpreter (≥ 3.10) | ~50MB |
|
||||
| pycapnp | `python3Packages.pycapnp` | capnp schema → Python types | small |
|
||||
| PyYAML | `python3Packages.pyyaml` | YAML config parsing | small |
|
||||
| jsonschema | `python3Packages.jsonschema` | JSON Schema validation | small |
|
||||
| click | `python3Packages.click` | CLI argument parsing | small |
|
||||
| pytest | `python3Packages.pytest` | Test framework | small |
|
||||
| ruff | `python3Packages.ruff` | Python formatter + linter | ~10MB |
|
||||
|
||||
All confirmed available in nixpkgs (unstable): pycapnp 2.2.3, pyyaml 6.0.3, jsonschema 4.26.0, click 8.3.1, pytest 9.0.3, ruff 0.15.20.
|
||||
|
||||
### Where to add
|
||||
|
||||
Two options:
|
||||
|
||||
**Option A: Add to `xce.nix`** (recommended). Every dev gets Python toolchain. Simpler — one flake, one shell. Python deps are ~60MB total, negligible vs current C toolchain.
|
||||
|
||||
**Option B: Separate `xceadmin.nix`**. Lighter shell for C-only devs. More maintenance — two nix files, two shells, cross-component workflows need manual env switching.
|
||||
|
||||
Recommend **Option A** — xceadmin is part of the standard toolchain (like xcecli), not an optional extra. All devs should be able to `xceadmin config build` without switching shells.
|
||||
|
||||
### New `nativeBuildInputs` block (addition to xce.nix)
|
||||
|
||||
```nix
|
||||
# Python — for xceadmin (admin config tool)
|
||||
python3
|
||||
python3Packages.pycapnp
|
||||
python3Packages.pyyaml
|
||||
python3Packages.jsonschema
|
||||
python3Packages.click
|
||||
python3Packages.pytest
|
||||
python3Packages.ruff
|
||||
```
|
||||
|
||||
### pycapnp build note
|
||||
|
||||
`pycapnp` wraps the capnproto C++ library at compile time (`pip install` compiles a C extension). Nix's `python3Packages.pycapnp` handles this correctly — it links against nix-provided `capnproto`. No additional `libcapnp-dev` or system package needed. This is one of the advantages of using nix pycapnp over `pip install pycapnp`.
|
||||
|
||||
### CI implications
|
||||
|
||||
- CI already uses nix dev shell — adding packages to `xce.nix` makes them available in CI automatically
|
||||
- `config build` golden-file tests need pycapnp for capnp encode (can't mock without schema)
|
||||
- `config push` mock tests need pycapnp + TLS (Python `ssl` stdlib, no extra dep)
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### config build (offline — pure data transform)
|
||||
|
||||
**Golden-file tests** (primary). Given a `configs/` directory, verify output matches expected:
|
||||
|
||||
```
|
||||
tests/
|
||||
├── fixtures/
|
||||
│ ├── minimal/ # hosts.yaml + queues.yaml only
|
||||
│ │ ├── configs/
|
||||
│ │ │ ├── hosts.yaml
|
||||
│ │ │ ├── queues.yaml
|
||||
│ │ │ └── schema/
|
||||
│ │ └── expected.bin # pre-built golden output
|
||||
│ ├── full/ # all 4 YAML files
|
||||
│ └── invalid/ # schema violations → expect error
|
||||
├── test_config_build.py # pytest: build → compare binary, verify SHA-256
|
||||
└── test_schema_validation.py # pytest: each schema violation case
|
||||
```
|
||||
|
||||
- Framework: `pytest` (standard Python, no deps beyond dev)
|
||||
- Zero external dependencies — no controller, no network
|
||||
- Pre-build golden files once; commit to repo
|
||||
- Schema validation: test each error case (missing required field, wrong type, invalid enum, reserved name `lost_and_found`)
|
||||
|
||||
### config push (online — needs controller)
|
||||
|
||||
**Option A: Mock controller** (unit/integration). Python TLS server that speaks enough capnp wire protocol to accept `OP_upgradeConfig` and return canned `UpgradeConfigReply`.
|
||||
|
||||
**Option B: Real controller** (E2E). Piggyback on existing `/xce-test-ha` cluster. After cluster is up, `xceadmin config push` a known config and verify controller accepts it.
|
||||
|
||||
- Option A for dev loop + CI (fast, no cluster setup)
|
||||
- Option B for acceptance (real SSL, real replication, real controller)
|
||||
|
||||
Recommended: Option A for spec tasks. Option B deferred to xce-test-ha integration after config-upgrade-snapshot lands.
|
||||
|
||||
### CI Integration
|
||||
|
||||
- `config build` tests: run in any Python CI (no C deps needed — pycapnp only for encode, can mock)
|
||||
- `config push` mock tests: need pycapnp + controller.capnp compiled for Python
|
||||
- Target: 80%+ coverage (project standard)
|
||||
|
||||
## Open Design Questions
|
||||
|
||||
Issues identified during 2026-07-07 review — need resolution before or during requirements generation.
|
||||
|
||||
### Critical (resolve before requirements)
|
||||
|
||||
| # | Question | Why it matters | Candidates |
|
||||
|---|----------|---------------|------------|
|
||||
| 1 | **YAML→capnp mapping incomplete** | `ClusterConfig` has 21 fields; 4 YAML files cover ~10. Remaining 11 (clusterName, port, candidates, heartbeat/interval timeouts, jobmgr/workermgr/qmgr/resmgr hash configs, xstrings, configGeneration) have no YAML source | (a) xceadmin fills defaults from controller's `get_default_conf()` values, (b) add a `cluster.yaml` for system-level fields, (c) accept CLI flags `--cluster-name`, `--port`, `--peers` |
|
||||
| 2 | **configGeneration** | Monotonic counter. **RESOLVED**: Controller uses it to detect new configs — same generation as current = no-op (nothing changed). xceadmin MUST read the current generation from the controller, increment it, and embed the incremented value in the binary before pushing. `xceadmin config build` sets it to 0 (offline-safe default). `xceadmin config push` auto-reads current generation, increments, rebuilds with new generation, then pushes. Admin can override with `--generation N`. Controller strictly rejects generations ≤ current (monotonic enforcement). | ✅ Decided: xceadmin-managed increment, controller-enforced monotonic |
|
||||
| 3 | **candidates field** | Controller peer list (`List(MasterCandidate)`). Currently passed as CLI args to `deploy-ha.sh`, not in YAML. xceadmin needs peer info for config build | (a) `--peers localhost:9991,localhost:9992,localhost:9993` CLI flag, (b) read from `hosts.yaml` host list, (c) separate `peers.yaml` |
|
||||
| 4 | **YAML key ≠ capnp field name** | YAML `groupname` → capnp `GroupConfig.name`, YAML `exclude_members` → capnp `exmembers`. Every field needs explicit mapping definition | Write a field-mapping table in design.md; auto-generate from capnp schema annotations if possible |
|
||||
| 5 | **Nested struct defaults** | `QueueConf` has 14 fields — YAML sets ~6. Remaining (attrs, status, schedDelay, resreq, preExec, postExec) need defaults | Use controller's `get_default_conf()` values from `myconf.c`; document which fields are operator-facing vs system-defaulted |
|
||||
|
||||
### Important (resolve during design)
|
||||
|
||||
| # | Question | Why it matters | Candidates |
|
||||
|---|----------|---------------|------------|
|
||||
| 6 | **Wire framing format** | Controller expects 8-byte uint64 size prefix (host byte order) + capnp payload. ~5 lines of Python but wrong byte order = silent reject | Document in design.md; mock controller test catches framing errors |
|
||||
| 7 | **TLS cert discovery** | xceadmin needs CA cert to verify controller TLS. Dev: `ha/data/xce.crt`. Production: configurable path | (a) `--ca-cert` flag, (b) `XCECERT` env var, (c) default `ha/data/xce.crt` for dev, require explicit for non-localhost |
|
||||
| 8 | **capnp schema compilation** | `capnp compile -o python controller.capnp` → `controller_capnp.py`. When and where? | (a) build time in xce-build.sh (committed output), (b) runtime in xceadmin startup (needs capnp binary), (c) nix derivation that produces the .py file |
|
||||
| 9 | **Python package layout** | Where does code live? Affects pip install, import paths, xce-build.sh | (a) `src/xceadmin/` alongside C code, (b) `tools/xceadmin/` with deploy-ha.sh, (c) top-level `xceadmin/` |
|
||||
| 10 | **existing cluster.bin provenance** | `ha/data/cluster.bin` is 8704 bytes, SHA 4986624f..., built June 21 by unknown method. Can't validate xceadmin output against it | Regenerate from known YAML inputs; use as golden-file test once xceadmin produces identical output for same inputs |
|
||||
|
||||
## Configuration Lifecycle & Rollback (Future Roadmap)
|
||||
|
||||
### Source of truth: Git-managed YAML
|
||||
|
||||
YAML config files (`configs/*.yaml`) are the authoritative source of truth. Operators manage them in git — version control, code review, audit trail are all free. `cluster.bin` is a **derived artifact** (like a compiled binary), never committed to git.
|
||||
|
||||
```
|
||||
git tag v1.2.0 → YAML at v1.2.0 → xceadmin config build → cluster.bin (deterministic)
|
||||
│
|
||||
xceadmin config push
|
||||
│
|
||||
▼
|
||||
controller cluster
|
||||
```
|
||||
|
||||
**Deterministic builds**: same YAML + same generation value + same xceadmin version → byte-identical `cluster.bin`. `configGeneration` defaults to 0 in offline builds (`xceadmin config build`) and is auto-incremented by `xceadmin config push` (read current from controller, +1). Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries.
|
||||
|
||||
### What git gives us (free)
|
||||
|
||||
| Capability | How |
|
||||
|-----------|-----|
|
||||
| Version history | `git log configs/` |
|
||||
| Diff between versions | `git diff v1.2.0 v1.3.0 -- configs/` |
|
||||
| Rollback source | `git checkout v1.2.0 -- configs/ && xceadmin config build` |
|
||||
| Audit trail | `git blame configs/queues.yaml` |
|
||||
| Code review | PR against `configs/` branch |
|
||||
|
||||
### What still needs building
|
||||
|
||||
Git manages the **source**. These parts need implementation:
|
||||
|
||||
| # | Capability | What it does | Owned by |
|
||||
|---|-----------|-------------|----------|
|
||||
| R1 | **Binary diff** | `xceadmin config diff cluster-v1.bin cluster-v2.bin` — human-readable diff of two built config binaries (decode both → compare field-by-field → pretty-print changes). Useful for pre-push review: "what will this config change actually do?" | admin-config-tool |
|
||||
| R2 | **Dry-run push** | `xceadmin config push --dry-run cluster.bin` — validate controller will accept the config (SHA check, version check) without committing. Safety net before real push | admin-config-tool |
|
||||
| R3 | **Config change snapshot** | After config upgrade, take an entity-bearing snapshot of the new configuration state. This anchors the config change in the snapshot chain — without it, rollback has no durable record of what the cluster looked like before/after the change. Controller-side: insert snapshot step into the `OP_upgradeConfig` pipeline (between step 5 "majority ACK" and step 6 "commit WAL_CONFIG_MARKER" in the current handler). | config-upgrade-snapshot |
|
||||
| R4 | **Controller version history** | Controller tracks last N config versions (tag + SHA + timestamp). `xceadmin config history` queries it. Needed for: "what config is the cluster running?" and "what was running before the last push?" Depends on R3 (snapshot chain provides durable version records). | config-upgrade-snapshot |
|
||||
| R5 | **Controller-side rollback** | `xceadmin config rollback` (or `xceadmin config push --rollback-to v1.2.0`) — push a previous config version. Controller-side concerns: (a) validate rollback target is safe (no running jobs reference deleted queues/hosts), (b) snapshot before rollback for forward-recovery, (c) bump configGeneration even on rollback (monotonic), (d) restart/activate the rolled-back config | config-upgrade-snapshot |
|
||||
| R6 | **Rollback safety validation** | Before rolling back, verify: (a) no running jobs use queues being deleted, (b) no active workers on hosts being removed, (c) resource definitions haven't changed in ways that break in-flight allocations. Controller-side check, client-side initiated | config-upgrade-snapshot |
|
||||
| R7 | **Config activation status** | `xceadmin config status` — show: current config tag, when applied, which nodes have it, any pending restart, snapshot status. Operator visibility into config rollout state | config-upgrade-snapshot |
|
||||
| R8 | **Config promotion pipeline** | Dev → staging → production config promotion. Same YAML source, different `--peers`/`--cluster-name` flags at build time. Git branches map to environments | admin-config-tool (future) |
|
||||
|
||||
### Controller-side pipeline: current vs target
|
||||
|
||||
```
|
||||
CURRENT (config_repl.c — exists today):
|
||||
xceadmin push → SHA verify → decode → gen++ → save → replicate → WAL marker → restart signal
|
||||
↑
|
||||
step 8 MISSING
|
||||
TARGET (after config-upgrade-snapshot):
|
||||
xceadmin push → SHA verify → decode → gen++ → save → replicate → [SNAPSHOT] → WAL marker → restart signal
|
||||
↑
|
||||
R3: entity-bearing
|
||||
snapshot of new config
|
||||
```
|
||||
|
||||
The snapshot insertion point matters: it goes AFTER replication (all standbys have the config) but BEFORE the WAL marker commit (so the snapshot is part of the committed state). This ensures a rollback target always has a consistent snapshot to restore from.
|
||||
|
||||
### Rollback workflow (end-to-end vision)
|
||||
|
||||
```
|
||||
# 1. Discover what went wrong
|
||||
xceadmin config diff cluster-current.bin cluster-suspect.bin
|
||||
# → "queues.normal.priority changed 10→1, queues.express removed"
|
||||
|
||||
# 2. Find the last good version
|
||||
xceadmin config history
|
||||
# → v1.2.0 (active), v1.1.0 (previous), v1.0.0 (initial)
|
||||
|
||||
# 3. Dry-run the rollback
|
||||
xceadmin config build --git-ref v1.1.0 -o cluster-v1.1.0.bin
|
||||
xceadmin config push --dry-run cluster-v1.1.0.bin
|
||||
# → "OK: controller will accept this config (generation 3→4)"
|
||||
|
||||
# 4. Execute rollback
|
||||
xceadmin config push cluster-v1.1.0.bin
|
||||
# → "Pushed (tag=v1.1.0, SHA=..., acked=3/3). Controller restarting."
|
||||
|
||||
# 5. Verify
|
||||
xceadmin config status
|
||||
# → "v1.1.0 active on 3/3 nodes"
|
||||
```
|
||||
|
||||
### Rollback safety: what can go wrong
|
||||
|
||||
| Scenario | Risk | Mitigation |
|
||||
|----------|------|------------|
|
||||
| Rollback deletes a queue | Running jobs on that queue become unmanageable | R5: controller refuses rollback while jobs reference deleted queues |
|
||||
| Rollback removes a host | Workers on that host lose config | R5: controller refuses rollback while active workers on removed hosts |
|
||||
| Rollback changes resource defs | In-flight allocations invalid | Controller fences: drain before rollback, or mark resources "changing" |
|
||||
| Rollback during network partition | Standbys miss the config push | Majority ACK required (existing OP_upgradeConfig behavior) |
|
||||
| Rapid rollback→forward→rollback | configGeneration churn, snapshot chain fragmentation | Rate-limit config changes (1 per N seconds); snapshot GC |
|
||||
|
||||
### What's in-scope for admin-config-tool vs deferred
|
||||
|
||||
| Capability | This spec | Why |
|
||||
|-----------|-----------|-----|
|
||||
| YAML source in git | ✅ (documentation) | Free — already how configs/ works |
|
||||
| Deterministic build | ✅ (core) | Same YAML → same binary (generation not set — controller-owned) |
|
||||
| Binary diff (R1) | ✅ (nice-to-have) | Pure Python, offline, high operator value |
|
||||
| Dry-run push (R2) | ✅ (nice-to-have) | Needs `ProbeConfigTag` RPC (already exists in controller) |
|
||||
| Config change snapshot (R3) | ❌ deferred | Controller-side — insert entity-bearing snapshot into OP_upgradeConfig pipeline; blocked on entity-bearing-snapshots |
|
||||
| Version history (R4) | ❌ deferred | Controller-side — depends on R3 snapshot chain |
|
||||
| Rollback (R5) | ❌ deferred | Controller-side — needs snapshot chain + restart |
|
||||
| Safety validation (R6) | ❌ deferred | Controller-side — needs job/worker state queries |
|
||||
| Config status (R7) | ❌ deferred | Controller-side — needs cluster-wide state tracking |
|
||||
| Promotion pipeline (R8) | ❌ deferred | Wraps git branches + xceadmin build — future enhancement |
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `config-upgrade-snapshot` (provides the client-side trigger that config-upgrade-snapshot's controller-side path responds to; R3 config-change-snapshot is the missing controller-side step — insert entity-bearing snapshot between replication and WAL marker commit in the `OP_upgradeConfig` pipeline)
|
||||
- **Adjacent**: `ha-term-safety` (the WAL_CONFIG_MARKER this tool's upload commits anchors on the Raft log via ha-term-safety's term-index infrastructure)
|
||||
- **Upstream blocker**: `entity-bearing-snapshots` — must land before R3 (config change snapshot) and R4 (version history) can be implemented
|
||||
- **Blocks**: nothing — xceadmin ships independently; the current controller pipeline (SHA verify → decode → gen++ → save → replicate → WAL marker → restart) already works without the snapshot step
|
||||
- **Future**: configuration lifecycle (R1–R8 above) — split across admin-config-tool (R1, R2, R8) and config-upgrade-snapshot (R3–R7)
|
||||
179
.kiro/specs/admin-config-tool/requirements.md
Normal file
179
.kiro/specs/admin-config-tool/requirements.md
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
XCE operators need a tool to build and deploy cluster configuration. Currently, YAML config files (`configs/{hosts,queues,resources,users}.yaml`) exist with JSON schemas for validation, but no tool compiles them into the `ClusterConfig` Cap'n Proto binary the controller expects. The controller-side `OP_upgradeConfig` handler exists (`config_repl.c` — receives binary, verifies SHA-256, pushes to standbys, awaits majority ACK), but no CLI sends it.
|
||||
|
||||
`xceadmin` is a Python CLI that fills this gap: `config build` compiles YAML into `cluster.bin`, `config push` sends it to a controller over TLS+capnp.
|
||||
|
||||
Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact — deterministic for the same inputs, never committed to git.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**:
|
||||
- YAML → `ClusterConfig` capnp binary compilation (`config build`)
|
||||
- TLS+capnp config deployment to controller (`config push`)
|
||||
- Auto primary discovery via controller redirect reply
|
||||
- JSON Schema validation of YAML inputs
|
||||
- Reserved name defense-in-depth (`"lost_and_found"`)
|
||||
- `xcecommon` shared Python package (capnp schema, wire protocol, config models)
|
||||
- Integration with xce-build and xce-deploy-ha skills
|
||||
- Nix dev shell: Python package dependencies
|
||||
- `config diff` — human-readable diff of two built config binaries (nice-to-have)
|
||||
- `config push --dry-run` — validate without committing (nice-to-have)
|
||||
|
||||
- **Out of scope**:
|
||||
- Controller-side config replication, snapshot, restart logic (owned by `config-upgrade-snapshot`)
|
||||
- Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`)
|
||||
- Performance tuning parameter extraction (separate work item)
|
||||
- xcecli Python rewrite (future, separate spec)
|
||||
- Controller config version history, rollback safety validation, config activation status (owned by `config-upgrade-snapshot`, see Future Roadmap in brief.md)
|
||||
|
||||
- **Adjacent expectations**:
|
||||
- **Controller `OP_upgradeConfig` handler** (`config_repl.c`) exists and is operational. On receiving `UpgradeConfigRequest{sha256, bytes}`:
|
||||
1. SHA-256 transit verification
|
||||
2. Cap'n Proto decode + reserved name check (`"lost_and_found"`)
|
||||
3. Auto-increment generation: `new_gen = current_gen + 1`, re-encode binary
|
||||
4. Save to disk, replicate to standbys, await majority ACK
|
||||
5. Commit `WAL_CONFIG_MARKER`, signal controller restart
|
||||
6. Return `UpgradeConfigReply{versionTag, ackedCount, error}`
|
||||
- **Controller redirect**: Standby returns `redirectHost`/`redirectPort` in the `RequestReply` union — xceadmin follows the redirect to reach the primary.
|
||||
- **ProbeConfigTag RPC** (`config_repl.c:1774`): Separate direct-capnp RPC (not in `Request` union) for reading current config tag. Used by `config push --dry-run` to validate before committing.
|
||||
- `controller.capnp` schema is the single source of truth for `ClusterConfig`, `Request`/`Reply` unions, and all OP codes.
|
||||
- `config-upgrade-snapshot` spec will extend the controller-side pipeline with: entity-bearing snapshots after config change, rollback safety validation, version history tracking, config activation status.
|
||||
- **Blocked controller-side gaps** (NOT owned by admin-config-tool):
|
||||
- Entity-bearing snapshot after config upgrade (blocked on `entity-bearing-snapshots` spec)
|
||||
- Config rollback safety (e.g., reject rollback while jobs reference deleted queues)
|
||||
- Config version history query (`xceadmin config history`)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Config Build
|
||||
|
||||
**Objective:** As an XCE operator, I want to compile YAML config files into a `ClusterConfig` capnp binary, so that I can produce a deployable cluster configuration from human-editable source files.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the operator runs `xceadmin config build <configs_dir>`, the system shall scan the directory for YAML config files (`hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`), validate each against its corresponding JSON schema from the `schema/` subdirectory, map the validated data to the `ClusterConfig` capnp struct, serialize it to binary, and write the result to `cluster.bin` in the current working directory.
|
||||
2. When the operator specifies `-o <path>`, the system shall write the output to the given path instead of the default `cluster.bin`.
|
||||
3. When the operator specifies `-o -`, the system shall write the binary output to stdout.
|
||||
4. After writing output, the system shall print the SHA-256 hash of the resulting binary to stderr.
|
||||
5. Where a YAML file is missing from the configs directory, the system shall treat the corresponding section as empty and continue building (missing files are not errors).
|
||||
6. Where the `schema/` subdirectory is missing from the configs directory, the system shall report an error and exit non-zero.
|
||||
7. Where the YAML content passes JSON Schema validation but maps to fields not present in the `ClusterConfig` capnp struct, the system shall warn the operator about unrecognized fields.
|
||||
|
||||
### Requirement 2: Config Push
|
||||
|
||||
**Objective:** As an XCE operator, I want to deploy a compiled `cluster.bin` to a controller cluster, so that the configuration takes effect on the running system.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the operator runs `xceadmin config push <cluster.bin>`, the system shall connect to the controller via TLS, send the binary as a capnp `Request` with `OP_upgradeConfig` opcode, `upgradeconfigrequestmsg` body (binary bytes + SHA-256 hash), framed with an 8-byte uint64 size prefix in host byte order, and display the result.
|
||||
2. When the operator specifies `--controller <host:port>`, the system shall connect to the given controller address instead of the default `localhost:9991`.
|
||||
3. After receiving the controller's reply, the system shall display the result to the operator: `versionTag` (the new generation number assigned by the controller), `ackedCount` (number of standbys that acknowledged), and any `error` message.
|
||||
4. When the controller indicates the config was committed (positive `ackedCount`), the system shall exit 0.
|
||||
5. If the controller reports an error (non-empty `error` field or zero `ackedCount`), the system shall display the error and exit non-zero.
|
||||
|
||||
### Requirement 3: Auto Primary Discovery
|
||||
|
||||
**Objective:** As an XCE operator, I want xceadmin to automatically find the primary controller, so that I don't need to know which node is currently primary.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the operator connects to any controller node (primary or standby), the system shall inspect the reply for `redirectHost` and `redirectPort` fields. If both are non-empty, the system shall reconnect to the redirect target and resend the request.
|
||||
2. When the redirect target itself redirects, the system shall follow at most 3 redirect hops before reporting an error.
|
||||
3. When a standby controller redirects the operator to the primary, the system shall display the redirect target to inform the operator where the config was actually pushed.
|
||||
4. When connecting to a controller and no primary has been elected yet, the system shall time out after 30 seconds and report "no primary available" to the operator.
|
||||
|
||||
### Requirement 4: Validation
|
||||
|
||||
**Objective:** As an XCE operator, I want validation errors caught early, so that invalid configuration never reaches a running cluster.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a YAML file fails JSON Schema validation, the system shall report the specific validation error with file name, line/field path, and the constraint violated, then exit non-zero.
|
||||
2. When a YAML file uses the reserved name `"lost_and_found"` for a queue or host, the system shall reject it with a clear error message explaining the reservation, even if it passes schema validation.
|
||||
3. When a YAML file contains duplicate entries (duplicate queue names, host names, resource names, group names), the system shall report the duplicates and exit non-zero.
|
||||
4. When a YAML file references an undefined entity (e.g., a queue references a host group that is not defined), the system shall warn the operator but allow the build to proceed (the controller performs final referential integrity checks at runtime).
|
||||
|
||||
### Requirement 5: xcecommon Shared Package
|
||||
|
||||
**Objective:** As a future Python tool author, I want a shared `xcecommon` package providing capnp schema loading, wire protocol, and config model types, so that I don't duplicate protocol code across tools.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The `xcecommon` package shall provide a capnp schema loader that reads `controller.capnp` and exposes the compiled Python types (`ClusterConfig`, `Request`, `Reply`, `UpgradeConfigRequest`, `UpgradeConfigReply`, and all nested types).
|
||||
2. The `xcecommon` package shall provide a wire protocol module that handles TLS connection, capnp message framing (8-byte uint64 size prefix in host byte order), send, and receive.
|
||||
3. The `xcecommon` package shall provide YAML→capnp mapping functions that convert validated YAML data to `ClusterConfig` capnp struct fields.
|
||||
4. Where the `xcecommon` package is installed for development, it shall be importable under the name `xcecommon` from any tool within the project.
|
||||
|
||||
### Requirement 6: Config Diff (Nice-to-Have)
|
||||
|
||||
**Objective:** As an XCE operator, I want to see what changed between two config binaries, so that I can review changes before pushing.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the operator runs `xceadmin config diff <cluster-v1.bin> <cluster-v2.bin>`, the system shall decode both binaries, compare them field-by-field, and display a human-readable summary of additions, removals, and modifications.
|
||||
2. The diff output shall group changes by section (queues, resources, hosts, users, limits, scalers) and show the specific fields that differ.
|
||||
3. When the two binaries are identical, the system shall report "no differences" and exit 0.
|
||||
|
||||
### Requirement 7: Dry-Run Push (Nice-to-Have)
|
||||
|
||||
**Objective:** As an XCE operator, I want to validate that the controller will accept a config before committing to the push.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the operator runs `xceadmin config push --dry-run <cluster.bin>`, the system shall connect to the controller, probe the generation, validate SHA and version compatibility, and report whether the config would be accepted, without actually committing the config to the cluster.
|
||||
2. When dry-run passes, the system shall report "OK: controller will accept this config (generation N→M)" and exit 0.
|
||||
3. When dry-run fails, the system shall report the specific rejection reason and exit non-zero.
|
||||
|
||||
### Requirement 8: Error Handling & Resilience
|
||||
|
||||
**Objective:** As an XCE operator, I want clear, actionable error messages for all failure modes, so that I can diagnose and fix configuration issues without reading controller logs.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. If the TLS connection to the controller fails, the system shall report "cannot connect to <host:port>: <reason>" and exit non-zero.
|
||||
2. If the controller returns a `Reply` with non-zero `status`, the system shall report the status code and any error text from the reply, then exit non-zero.
|
||||
3. If the capnp binary is corrupted or cannot be decoded, the system shall report "invalid cluster.bin: <reason>" and exit non-zero.
|
||||
4. If the `controller.capnp` schema file is not found at the expected path, the system shall report the missing path and suggest checking the installation.
|
||||
5. If the operator interrupts the push (SIGINT), the system shall clean up the TLS connection and exit gracefully with a "cancelled" message.
|
||||
6. The system shall time out after 30 seconds on any network operation and report "timeout connecting to <host:port>".
|
||||
|
||||
### Requirement 9: Skill & Harness Integration
|
||||
|
||||
**Objective:** As a developer, I want xceadmin integrated into the existing development toolchain, so that building, deploying, and testing XCE clusters uses consistent tooling.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the developer runs `/xce-build xceadmin`, the system shall make the `xceadmin` CLI available on PATH and run the full test suite, reporting pass/fail.
|
||||
2. When the developer runs `/xce-build xceadmin release`, the system shall install `xceadmin` for production use (pinned, non-editable).
|
||||
3. When the developer runs `/xce-deploy-ha` with the `--build-config` flag, the system shall invoke `xceadmin config build configs/ -o ha/data/cluster.bin` as a precondition before deploying the cluster.
|
||||
4. When `cluster.bin` is missing from `ha/data/` during deploy, `/xce-deploy-ha` shall run `xceadmin config build` automatically (same as `--build-config`).
|
||||
5. When `cluster.bin` already exists and `--build-config` is not passed, `/xce-deploy-ha` shall use the existing binary as-is (backward compatible).
|
||||
6. The nix dev shell (`nix develop ./src`) shall include all Python dependencies required by xceadmin after the nix environment is updated (see Nix Environment section in brief.md).
|
||||
|
||||
### Requirement 10: Deterministic Builds
|
||||
|
||||
**Objective:** As an XCE operator, I want config builds to be deterministic, so that I can reproduce any historical config binary from its YAML source.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When `xceadmin config build` is run twice with the same YAML inputs and same xceadmin version, the system shall produce byte-identical `cluster.bin` output.
|
||||
2. The system shall not embed build timestamps, hostnames, usernames, or any other non-deterministic data in the output binary.
|
||||
3. When different operators on different machines build the same YAML with the same xceadmin version, the output shall be byte-identical.
|
||||
|
||||
## Requirements Summary
|
||||
|
||||
| # | Requirement | Priority | Phase |
|
||||
|---|------------|----------|-------|
|
||||
| 1 | Config Build | **Core** | Implementation |
|
||||
| 2 | Config Push | **Core** | Implementation |
|
||||
| 3 | Auto Primary Discovery | **Core** | Implementation |
|
||||
| 4 | Validation | **Core** | Implementation |
|
||||
| 5 | xcecommon Package | **Core** | Implementation |
|
||||
| 6 | Config Diff | Nice-to-Have | Follow-up |
|
||||
| 7 | Dry-Run Push | Nice-to-Have | Follow-up |
|
||||
| 8 | Error Handling | **Core** | Implementation |
|
||||
| 9 | Skill & Harness | **Core** | Implementation |
|
||||
| 10 | Deterministic Builds | **Core** | Implementation |
|
||||
23
.kiro/specs/admin-config-tool/spec.json
Normal file
23
.kiro/specs/admin-config-tool/spec.json
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"feature_name": "admin-config-tool",
|
||||
"created_at": "2026-07-06T01:30:00Z",
|
||||
"updated_at": "2026-07-07T08:57:00Z",
|
||||
"language": "en",
|
||||
"phase": "requirements-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": []
|
||||
}
|
||||
200
.kiro/specs/scheduler-framework/brief.md
Normal file
200
.kiro/specs/scheduler-framework/brief.md
Normal file
|
|
@ -0,0 +1,200 @@
|
|||
# Brief: scheduler-framework
|
||||
|
||||
## Problem
|
||||
The XCE scheduler has a plugin architecture but three critical gaps prevent it from being a real framework:
|
||||
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:48` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation.
|
||||
2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation.
|
||||
3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
|
||||
|
||||
Operators need configurable scheduling policies and parallel job support. The scheduler must generate multiple candidate allocation plans and try them in preference order.
|
||||
|
||||
## Current State
|
||||
|
||||
### What exists
|
||||
- **Plugin framework** (`plugin.h`, `plugin.c`): `xce_sched_plugin_t` with 9 hooks + `job_elect_fn` / `job_alloc_fn`. Plugins registered in key-value map by name.
|
||||
- **FCFS job picker** (`fcfs.c`): dedicated `jobpicker` thread. Iterates queues by priority, jobs within queue by (priority desc, queue_time asc). Communicates picked jobid via `jbuf` cbuf.
|
||||
- **Seqjob allocator** (`seqjob_allocator.c`): handles `maxTasks == 1`. Checks shared resources → collects workers → tries one host → returns first match.
|
||||
- **Parjob allocator stub** (`parjob_allocator.c`): registered but `my_alloc()` is empty.
|
||||
- **Resource request types** (`resreq.h`): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds).
|
||||
- **Select expression AST** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC.
|
||||
- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session.
|
||||
- **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next.
|
||||
|
||||
### What's missing
|
||||
- Configurable job picker: the plugin name is a hardcoded string literal
|
||||
- Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
|
||||
- Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
|
||||
- Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
1. **Configurable job selection**: operator can set `job_pick_plugin` in scheduler config to swap policies (FCFS, priority, custom) without recompilation. Plugin interface contract is documented (cancel-safe, session-aware).
|
||||
2. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
|
||||
3. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
|
||||
4. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
|
||||
5. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Approach A: Incremental Enhancement (rejected)
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Add `job_pick_plugin` config field, same as chosen approach |
|
||||
| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation — all plans materialized before first try |
|
||||
| **Iterator** | No new abstraction. Simple for-loop over plan array, inner for-loop over hosts |
|
||||
| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state, same as chosen approach |
|
||||
|
||||
**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces (e.g., 64 slots on 128 hosts = thousands of plans). No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.
|
||||
|
||||
### Approach B: Iterator-Driven Allocation Architecture ★ Chosen
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Configurable plugin name via `job_pick_plugin` field |
|
||||
| **Iterator abstraction** | `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy, cancel-safe, composable |
|
||||
| **Parallel jobs** | Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 |
|
||||
| **Host iterator** | Per-plan lazy candidate host iteration using existing select expression eval |
|
||||
| **Shared resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` |
|
||||
|
||||
**Chosen because:** Lazy generation avoids memory pressure. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. Clean separation of concerns. Each iterator implementation is independently testable.
|
||||
|
||||
### Approach C: Full Scheduler Framework (deferred)
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Policy chain — multiple plugins score/veto jobs. Composable: priority + fair-share + preemption |
|
||||
| **Iterator** | Triple composition: job iterator → plan iterator → host iterator |
|
||||
| **Resreq registry** | Named primary resreqs registered globally, referenced by name, reference-counted |
|
||||
| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin, topology-aware |
|
||||
| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT with well-defined transitions |
|
||||
|
||||
**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's iterator abstraction is designed to compose upward — the plan iterator and host iterator are building blocks that a future job iterator or policy chain can consume. The template registry can layer on top without breaking the iterator interfaces.
|
||||
|
||||
### Decision rationale
|
||||
|
||||
Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). Approach C's additional layers (policy chains, resreq registry, state machine) are deferred until real use cases emerge — the iterator interfaces are designed to compose upward when needed.
|
||||
|
||||
## Approach
|
||||
|
||||
**Iterator-Driven Allocation Architecture** (Approach B):
|
||||
|
||||
### Core abstractions
|
||||
|
||||
```
|
||||
xce_alloc_plan_iter_t xce_host_iter_t
|
||||
┌─────────────────────┐ ┌──────────────────────┐
|
||||
│ init(resreq, sched) │ │ init(plan, workers) │
|
||||
│ next() → plan │───────▶│ next() → worker │
|
||||
│ fini() │ │ fini() │
|
||||
└─────────────────────┘ └──────────────────────┘
|
||||
```
|
||||
|
||||
- **Plan iterator**: owned by allocator plugin (`parjob_allocator` or `seqjob_allocator`). Generates `(nhosts, slots_per_host, host_filter)` plans lazily.
|
||||
- **Host iterator**: per-plan. Wraps `sched_collect_workers()` + select expression filtering into a lazy interface. Each `next()` returns the next candidate host or NULL.
|
||||
|
||||
### Scheduling loop (modified)
|
||||
|
||||
```
|
||||
for each elected job:
|
||||
plan_iter = job.allocator.make_plan_iter(resreq)
|
||||
while (plan = plan_iter.next()):
|
||||
host_iter = make_host_iter(plan, all_workers)
|
||||
while (host = host_iter.next()):
|
||||
if try_alloc(host, plan) succeeds:
|
||||
dispatch(job, host, plan)
|
||||
goto next_job
|
||||
host_iter.advance()
|
||||
host_iter.fini()
|
||||
plan_iter.fini()
|
||||
pend(job, reason)
|
||||
```
|
||||
|
||||
### Configurable job picker
|
||||
|
||||
- Add `char* job_pick_plugin` to `xce_scheduler_t` (default `"xce_sched_fcfs"`)
|
||||
- `sched_job_picker()` reads this field instead of hardcoded string
|
||||
- Set via scheduler config or command-line flag
|
||||
- Plugin interface contract: `job_elect_fn` must be cancel-safe (respond to `pthread_cancel`), session-aware (reset state on `end_session`), and communicate via `jbuf` cbuf
|
||||
|
||||
### Unsatisfiable resreq cache
|
||||
|
||||
- Per scheduling session (`xce_scheduler_t`), cleared on `begin_session`
|
||||
- Key: hash of `{select_string, rusage_string, user, queue, project}`
|
||||
- Before full allocation: quick lookup. If present → skip to next job immediately
|
||||
- Populated when a job's plan iterator exhausts all plans without success
|
||||
|
||||
### What stays unchanged
|
||||
- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved
|
||||
- `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot
|
||||
- `fcfs.c` — continues as default job picker, no changes needed
|
||||
- Select expression system — no changes
|
||||
- COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
|
||||
- Controller communication (handler.c opcodes, decision cbuf) — unchanged
|
||||
|
||||
## Scope
|
||||
|
||||
### In
|
||||
- `xce_alloc_plan_iter_t` interface (header + implementation)
|
||||
- `xce_host_iter_t` interface (header + implementation)
|
||||
- `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
|
||||
- Configurable job picker: `job_pick_plugin` field + config wiring
|
||||
- Session-level unsatisfiable resreq fingerprint cache
|
||||
- Plugin interface contract documentation (cancel-safe, session-aware)
|
||||
- Unit tests for plan generation, host iteration, cache hit/miss
|
||||
- Integration test: multi-slot parallel job allocation end-to-end
|
||||
|
||||
### Out
|
||||
- New job selection policies (priority-only, fair-share, preemption, backfill) — framework enables them but they ship separately
|
||||
- Resreq template registry (named/shared resreq objects) — future enhancement
|
||||
- Select expression language changes
|
||||
- Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item)
|
||||
- Compound/ALTER resreq allocation changes
|
||||
- GPU/affinity-aware allocation (pluggable later via allocator plugins)
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
The work naturally splits into these implementation phases (single spec, sequential):
|
||||
|
||||
1. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
|
||||
2. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
|
||||
3. **Configurable job picker** — add `job_pick_plugin` config, document plugin contract
|
||||
4. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop
|
||||
5. **Integration + testing** — end-to-end multi-slot job allocation, cache behavior, config swapping
|
||||
|
||||
## Out of Boundary
|
||||
- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec
|
||||
- Resreq template sharing across jobs (future: `resreq_registry` spec)
|
||||
- Commit barrier wiring for scheduler decisions (separate hardening item in roadmap)
|
||||
- Controller-side changes (the scheduler runs independently, receives updates via existing opcodes)
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
### Upstream (dependencies)
|
||||
- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed
|
||||
- Existing select expression evaluator (`resreq.h` eval/Gc) — NO changes needed
|
||||
- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed
|
||||
- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — minimal additions
|
||||
|
||||
### Downstream (future consumers)
|
||||
- **New job selection policies** — priority, fair-share, preemption plugins register via the configurable picker
|
||||
- **Affinity/GPU allocators** — register as allocator plugins, implement plan iterator for topology-aware placement
|
||||
- **Commit barrier wiring** — scheduling decisions flow through majority commit (separate hardening item)
|
||||
- **Resreq template registry** — named resreqs shared across jobs, reference-counted
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
### Extends
|
||||
- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same `xce_scheduler_t` state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.
|
||||
|
||||
### Adjacent
|
||||
- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.
|
||||
- **worker-join-refactor** — worker state management. Allocation consumes worker resource availability pushed via the existing join/update path.
|
||||
|
||||
## Constraints
|
||||
- **C (gnu17)** — no C++, no external libraries beyond existing (capnp, zlib, pthreads)
|
||||
- **Non-blocking in plugin hooks** — allocation happens on allocator thread, not iomgr. No new blocking concerns.
|
||||
- **Cancel-safe** — job picker thread is `pthread_cancel`-able. Iterator `next()` calls must have cancellation points.
|
||||
- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session`.
|
||||
- **Backward compatible** — existing FCFS + seqjob behavior unchanged. New functionality is additive.
|
||||
- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded).
|
||||
|
|
@ -1,7 +1,15 @@
|
|||
# anatomy.md
|
||||
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-07-05T07:39:59.974Z
|
||||
> Files: 600 tracked | Anatomy hits: 0 | Misses: 0
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-07-09T10:29:48.021Z
|
||||
> Files: 620 tracked | Anatomy hits: 0 | Misses: 0
|
||||
|
||||
## ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/5e55cf26-d05d-4a2e-8196-d2c70e0c063c/scratchpad/
|
||||
|
||||
- `bench-v3.sh` — bench-v3.sh — robust throughput measurement. Wave-submit N jobs (fresh (~843 tok)
|
||||
- `bench-v4.sh` — bench-v4.sh — throughput measurement, fixed to count DONE only among THIS (~1083 tok)
|
||||
- `factstream-throughput-bench-v2.sh` — factstream-throughput-bench-v2.sh — task 7.4 (raft-apply-decoupling) fix (~1576 tok)
|
||||
- `factstream-throughput-bench.sh` — factstream-throughput-bench.sh — task 7.4 (raft-apply-decoupling) throughput (~1750 tok)
|
||||
- `run_all_groups.sh` (~301 tok)
|
||||
|
||||
## ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/
|
||||
|
||||
|
|
@ -15,8 +23,12 @@
|
|||
|
||||
## ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/
|
||||
|
||||
- `MEMORY.md` — Memory Index (~1681 tok)
|
||||
- `MEMORY.md` — Memory Index (~1906 tok)
|
||||
- `project_bug1306_hook_rc_clobber.md` (~693 tok)
|
||||
- `project_bug1417_vote_index_mapping.md` — Bug-1417: Real-vote reply index mapping mismatch (~784 tok)
|
||||
- `project_bug1418_term_desync_apply_divergent.md` — Bug-1418: Term desync + apply-divergence promotion block → chaotic liveness stall (~743 tok)
|
||||
- `project_bug1419_wal_dup_submit_false_divergent.md` — Declares occurrence (~955 tok)
|
||||
- `project_bug847_wal_batch_payload_loss.md` (~1213 tok)
|
||||
- `project_raft_election_resonance.md` (~556 tok)
|
||||
- `reference_nonoverlay_fs_soak.md` (~371 tok)
|
||||
|
||||
|
|
@ -24,6 +36,10 @@
|
|||
|
||||
- `ctrl-dbg.sh` — bug-1283: run the NON-ASAN DEBUG (-O0 -g3) xce_controller under gdb -batch; on (~231 tok)
|
||||
|
||||
## ../../xce/.claude/worktrees/agent-a0dfba18c450e4f5f/src/controller/testsuite/
|
||||
|
||||
- `test.c` — include <argp.h> (~28571 tok)
|
||||
|
||||
## ./
|
||||
|
||||
- `.clang-format` (~226 tok)
|
||||
|
|
@ -217,16 +233,16 @@
|
|||
|
||||
## .claude/skills/xce-build/
|
||||
|
||||
- `SKILL.md` — build xce source code (~380 tok)
|
||||
- `SKILL.md` — build xce source code (~985 tok)
|
||||
|
||||
## .claude/skills/xce-deploy-ha/
|
||||
|
||||
- `SKILL.md` — XCE HA Cluster Deployment (~1345 tok)
|
||||
- `SKILL.md` — XCE HA Cluster Deployment (~1678 tok)
|
||||
|
||||
## .claude/skills/xce-test-ha/
|
||||
|
||||
- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then (~18920 tok)
|
||||
- `SKILL.md` — XCE HA Failover Soak Test (~4553 tok)
|
||||
- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then (~19673 tok)
|
||||
- `SKILL.md` — XCE HA Failover Soak Test (~5002 tok)
|
||||
- `soak-verdict-selftest.sh` — soak-verdict-selftest.sh — deterministic unit test for the verdict parser in (~1857 tok)
|
||||
- `soak-verdict.sh` — soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh. (~1383 tok)
|
||||
|
||||
|
|
@ -280,6 +296,12 @@
|
|||
|
||||
- `.gitignore` — Git ignore rules (~3 tok)
|
||||
|
||||
## .kiro/specs/admin-config-tool/
|
||||
|
||||
- `brief.md` — Brief: admin-config-tool (~9037 tok)
|
||||
- `requirements.md` — Requirements Document (~3532 tok)
|
||||
- `spec.json` (~138 tok)
|
||||
|
||||
## .kiro/specs/config-upgrade-snapshot/
|
||||
|
||||
- `brief.md` — Brief: config-upgrade-snapshot (~923 tok)
|
||||
|
|
@ -468,12 +490,12 @@
|
|||
## .kiro/specs/raft-apply-decoupling/
|
||||
|
||||
- `brief.md` — Brief: raft-apply-decoupling (~2239 tok)
|
||||
- `design.md` — Design Document: raft-apply-decoupling (~9210 tok)
|
||||
- `design.md` — Design Document: raft-apply-decoupling (~9566 tok)
|
||||
- `facts-in-raft.md` — Design Note: Routing Facts Through Raft (apply-after-commit for RETAIN events) (~5569 tok)
|
||||
- `requirements.md` — Requirements Document (~4629 tok)
|
||||
- `research.md` — Research & Design Decisions (~4600 tok)
|
||||
- `spec.json` (~129 tok)
|
||||
- `tasks.md` — Implementation Plan (~20719 tok)
|
||||
- `tasks.md` — Implementation Plan (~23836 tok)
|
||||
|
||||
## .kiro/specs/raft-commit-safety/
|
||||
|
||||
|
|
@ -544,6 +566,10 @@
|
|||
- `spec.json` (~230 tok)
|
||||
- `tasks.md` — Tasks: scheduler-controller-msg-gap (~4650 tok)
|
||||
|
||||
## .kiro/specs/scheduler-framework/
|
||||
|
||||
- `brief.md` — Brief: scheduler-framework (~3335 tok)
|
||||
|
||||
## .kiro/specs/worker-join-refactor/
|
||||
|
||||
- `design.md` — Design Document: worker-join-refactor (~4709 tok)
|
||||
|
|
@ -568,7 +594,7 @@
|
|||
- `ha-async-io.md` — HA Protocol & Async I/O (~3225 tok)
|
||||
- `persistence.md` — WAL & Persistence Durability (~2127 tok)
|
||||
- `product.md` — Product Overview (~1246 tok)
|
||||
- `roadmap.md` — Roadmap (~8482 tok)
|
||||
- `roadmap.md` — Roadmap (~10221 tok)
|
||||
- `security.md` — Security Standards (~804 tok)
|
||||
- `structure.md` — Project Structure (~1170 tok)
|
||||
- `tech.md` — Technology Stack (~1155 tok)
|
||||
|
|
@ -903,21 +929,21 @@
|
|||
## src/controller/
|
||||
|
||||
- `CLAUDE.md` — XCE Controller (~3373 tok)
|
||||
- `cluster.c` — include "config.h" (~18098 tok)
|
||||
- `cluster.c` — include "config.h" (~18444 tok)
|
||||
- `commit.c` — Declares is (~12825 tok)
|
||||
- `config_repl.c` — define _GNU_SOURCE (~28945 tok)
|
||||
- `config_repl.c` — define _GNU_SOURCE (~29886 tok)
|
||||
- `config_repl.h` — pragma once (~6155 tok)
|
||||
- `ha.c` — define _GNU_SOURCE (~102306 tok)
|
||||
- `ha.h` — ifndef HA_H (~26001 tok)
|
||||
- `job.c` — include <errno.h> (~16424 tok)
|
||||
- `ha.c` — define _GNU_SOURCE (~103039 tok)
|
||||
- `ha.h` — ifndef HA_H (~26377 tok)
|
||||
- `job.c` — include <errno.h> (~18619 tok)
|
||||
- `main.c` — include <argp.h> (~3035 tok)
|
||||
- `meson.build` (~1580 tok)
|
||||
- `myapp.c` — define _GNU_SOURCE (~5215 tok)
|
||||
- `persist.c` — define _GNU_SOURCE (~60786 tok)
|
||||
- `persist.c` — define _GNU_SOURCE (~63473 tok)
|
||||
|
||||
## src/controller/include/
|
||||
|
||||
- `cluster.h` — if !defined(_CLUSTER_H_) (~2610 tok)
|
||||
- `cluster.h` — if !defined(_CLUSTER_H_) (~2740 tok)
|
||||
- `commit.h` — pragma once (~3635 tok)
|
||||
|
||||
## src/controller/testsuite/
|
||||
|
|
@ -927,19 +953,21 @@
|
|||
- `snapdiff.h` — pragma once (~569 tok)
|
||||
- `snapverify.c` — define _GNU_SOURCE (~2505 tok)
|
||||
- `snapverify.h` — pragma once (~645 tok)
|
||||
- `test_apply_committed_entry.c` — Declares directly (~4141 tok)
|
||||
- `test_apply_correctness_verification.c` — Task 5.1-5.3: post-replay apply verdict (CLEAN/DIVERGENT), routes through existing loud-fence; 11 tests incl. explicit stays-alive pin + fence-clear-via-restart + direct persist_clear_recovery_unrecoverable coverage (~17000 tok)
|
||||
- `test_apply_committed_entry.c` — Declares directly (~4253 tok)
|
||||
- `test_apply_correctness_verification.c` — Declares driving (~14297 tok)
|
||||
- `test_apply_pump_integration.c` — Declares hook (~16913 tok)
|
||||
- `test_apply_pump.c` — Task 6.1 ordered apply pump tests (4, pool=NULL) + task 6.2 round-2 bug-1415 regression `test_apply_pump_late_pending_apply_registration_self_heals` (5th test, own tpr_* fixture with a REAL multi-vcpu vtaskmgr_t via commit_set_pool, forces apply-pump-wins-race via a barrier) (~17700 tok)
|
||||
- `test_apply_reapply_idempotency.c` — Declares xce_job_t (~4001 tok)
|
||||
- `test_apply_reapply_idempotency.c` — Declares xce_job_t (~4026 tok)
|
||||
- `test_apply_seam.c` (~2936 tok)
|
||||
- `test_apply_seam.c` — raft-apply-decoupling task 2.1: calls apply_job_start/finish/fail/update_transition() directly on a bare jobmgr+job_t (no cluster_t), bypassing on_job_start/finish/fail, to prove the extracted guard cores work outside the receipt path (~2900 tok)
|
||||
- `test_commit_barrier_reply.c` — Declares uint8_t (~38815 tok)
|
||||
- `test_commit_barrier_reply.c` — Declares uint8_t (~41183 tok)
|
||||
- `test_commit_durability.c` — define _GNU_SOURCE (~31376 tok)
|
||||
- `test_commit_resolve.c` (~14018 tok)
|
||||
- `test_config_repl.c` — define _GNU_SOURCE (~42756 tok)
|
||||
- `test_convergence_verification.c` — define _GNU_SOURCE (~16987 tok)
|
||||
- `test_demotion_convergence.c` — define _GNU_SOURCE (~12767 tok)
|
||||
- `test_e2e_checkpoint.c` — define _GNU_SOURCE (~3544 tok)
|
||||
- `test_e2e_gc.c` — define _GNU_SOURCE (~1678 tok)
|
||||
- `test_e2e_hooks.c` — define _GNU_SOURCE (~2184 tok)
|
||||
- `test_egress_filter.c` — Declares xce_job_t (~5072 tok)
|
||||
- `test_ha_core_safety.c` — Declares char (~36309 tok)
|
||||
|
|
@ -948,14 +976,17 @@
|
|||
- `test_ha_rdhup_integration.c` — Declares char (~7341 tok)
|
||||
- `test_ha_vep_teardown.c` (~6218 tok)
|
||||
- `test_ha_vote_gate.c` — define _GNU_SOURCE (~14165 tok)
|
||||
- `test_ha_vote_send.c` — define _GNU_SOURCE (~6961 tok)
|
||||
- `test_persist_checkpoint.c` — define _GNU_SOURCE (~2415 tok)
|
||||
- `test_persist_hooks.c` — define _GNU_SOURCE (~4305 tok)
|
||||
- `test_persist_recovery.c` — define _GNU_SOURCE (~3778 tok)
|
||||
- `test_persist_snapshot_entities.c` — unsigned: queue (~14897 tok)
|
||||
- `test_persist_truncate.c` — define _GNU_SOURCE (~5031 tok)
|
||||
- `test_replay_apply_seam.c` — Declares through (~3707 tok)
|
||||
- `test_snapdiff.c` — define _GNU_SOURCE (~1033 tok)
|
||||
- `test_snapverify.c` — define _GNU_SOURCE (~2008 tok)
|
||||
- `test_walcompare.c` — define _GNU_SOURCE (~6845 tok)
|
||||
- `test.c` — include <argp.h> (~29726 tok)
|
||||
- `test.c` — include <argp.h> (~30294 tok)
|
||||
- `walcompare.c` — define _GNU_SOURCE (~5176 tok)
|
||||
- `walcompare.h` — pragma once (~1474 tok)
|
||||
- `waldump.c` — define _GNU_SOURCE (~5404 tok)
|
||||
|
|
@ -967,8 +998,8 @@
|
|||
- `delay.h` — if !defined(_DELAY_H_) (~414 tok)
|
||||
- `myconf.h` — if !defined(_MYCONF_H_) (~560 tok)
|
||||
- `netio.h` — if !defined(_NETIO_H_) (~2313 tok)
|
||||
- `persist.h` — pragma once (~20431 tok)
|
||||
- `typemap.h` — if !defined(_TYPEMAP_H_) (~4717 tok)
|
||||
- `persist.h` — pragma once (~21365 tok)
|
||||
- `typemap.h` — if !defined(_TYPEMAP_H_) (~5116 tok)
|
||||
- `wal_types.h` — pragma once (~1996 tok)
|
||||
|
||||
## src/lib/
|
||||
|
|
@ -976,25 +1007,26 @@
|
|||
- `delay.c` — include <errno.h> (~1520 tok)
|
||||
- `jobj.c` — include <errno.h> (~4867 tok)
|
||||
- `netio.c` — include <arpa/inet.h> (~8371 tok)
|
||||
- `request.c` — include <errno.h> (~15919 tok)
|
||||
- `request.c` — include <errno.h> (~16804 tok)
|
||||
- `vtaskmgr.c` — include <errno.h> (~3822 tok)
|
||||
|
||||
## src/lib/raft/
|
||||
|
||||
- `raft_sim.c` — ifdef XCE_TESTING (~28490 tok)
|
||||
- `raft.h` — ifndef RAFT_H (~17581 tok)
|
||||
- `raft.c` — include "raft.h" (~9701 tok)
|
||||
- `raft.h` — ifndef RAFT_H (~17836 tok)
|
||||
|
||||
## src/lib/testsuite/
|
||||
|
||||
- `test_ctrlstatus.c` (~4821 tok)
|
||||
- `test_raft.c` — include <errno.h> (~98108 tok)
|
||||
- `test_wal_wire.c` — include <errno.h> (~4238 tok)
|
||||
- `test_wal_wire.c` — include <errno.h> (~4204 tok)
|
||||
|
||||
## src/schema/
|
||||
|
||||
- `controller.capnp` — Declares OPCode (~9344 tok)
|
||||
- `controller.capnp` — Declares OPCode (~9720 tok)
|
||||
|
||||
## src/tools/
|
||||
|
||||
- `test-commit-durability.sh` — test-commit-durability.sh — bug-661 reproduction: committed jobs lost across failover. (~9574 tok)
|
||||
- `xce-build.sh` (~1395 tok)
|
||||
- `xce-build.sh` (~1400 tok)
|
||||
|
|
|
|||
|
|
@ -7,8 +7,12 @@
|
|||
## User Preferences
|
||||
|
||||
<!-- How the user likes things done. Code style, tools, patterns, communication. -->
|
||||
- Run soak tests with specific round counts and deploy directories for precise regression tracking
|
||||
- Prefer parallel agent runs for independent analysis work; sequential for dependent tasks
|
||||
- Keep test logs for analysis, don't skip intermediate output even if a test hangs
|
||||
|
||||
## Key Learnings
|
||||
- (2026-07-08, HA soak test sequential vs chaotic modes) Sequential mode soak validates clean failover lifecycle: 10 rounds, 36 jobs, zero loss, all 3 nodes led primary — deterministic validation of promotion + catch-up + reapply paths working as designed. Chaotic mode deliberately overlaps kills across election/catch-up windows to surface edge cases: rounds 1-4 pass verification, round 5 hits primary-discovery lag (bug-854 root-caused earlier, excluded from verdict), rounds 6-7 stall from election convergence lag + duplicate-node-guard (bug-1418, pre-existing guard). The election stall in chaotic round 6 is likely compound: (a) tight overlapping kill cadence outpaces election+restart timing (heartbeat-timeout bounds ~500ms but nix shell spinup adds latency), (b) a node restarted but not fully converged rejoins before the prior kill round's primary finishes its term tenure. Sequential mode PASS verdict is solid baseline. Chaotic mode needs either slower kill spacing OR a real-fs host deploy (current sandbox has overlayfs fdatasync EIO bug-924) to distinguish fault-tolerance (still holds) from environment-induced timing collapse.
|
||||
- (2026-07-04, raft-apply-decoupling task 6.1 round 2 — job-state merge-staleness fix) Round 1's `apply_committed_entry` wired the live pump but left a real corruption: `record_job_fail_fact`/`record_job_update_fact` (job.c) precomputed a FINAL absolute state at RECEIPT time (a scratch-copy merge for FAIL, an inline backstop+PDONE merge for postdone) and baked it into `wal_jobstate_t.state`; under deferred apply, a second fact for the same job (e.g. FINISH) could apply BETWEEN receipt and this fact's own apply, making the baked snapshot stale — a POST_EXEC-stage FAIL received while a job is still `RUN` (FINISH not yet applied) bakes `RUN|EXIT|PERR`, and when FINISH applies first (`DONE`) then FAIL's blind absolute overwrite clobbers it back to `RUN|EXIT|PERR` — DONE silently lost. FIX PATTERN: `wal_jobstate_t._pad` (previously always-zero, unused) is repurposed as a discriminator — `_pad==0` still means "legacy absolute state, safe" (job_ctrl's mark_job_fail/mark_job_update paths, which mutate+capture state synchronously under one lock, no staleness window); a nonzero `_pad` (`WAL_JOB_FAIL_STAGE_FLAG|stage` for FAIL, `WAL_JOB_UPDATE_POSTDONE_MARK` for postdone) marks "recompute at apply time against ACTUAL current state" — `apply_committed_entry` (persist.c) dispatches on this bit per WAL type, calling the REAL `apply_job_fail_transition`/new `apply_job_postdone_transition` (job.c, mirrors the existing guard-core family) instead of `apply_job_update_transition`'s blind overwrite. New low-level `persist_submit_job_fact_ex(p, type, jobid, state, pad)` added as the primitive; `persist_submit_job_fact` becomes a pad=0 thin wrapper — ZERO existing call sites (including ~20 in test_commit_barrier_reply.c) needed touching. GENERAL PATTERN for "wire format has 2 producers, one safe one not": don't change the safe producer's shape — add a discriminator field/flag so the consumer can tell them apart, and make "0"/"absent" mean exactly the old/safe behavior so every existing writer keeps working unchanged.
|
||||
- (2026-07-04, same task) Fixing a wire-format bug INSIDE a task's own scope can break pre-existing tests that assert the wire format directly (not just behavior) — 2 tests in test_commit_barrier_reply.c (`test_guard_forward_update_on_terminal_preserved`, `test_guard_fail_after_finish_preserves_perr_merge`) read the raw on-disk `wal_jobstate_t.state` via a helper (`cbrp_count_jobstate_entries`) and asserted DONE/PDONE/PERR bits directly on it — exactly the field this fix stops populating for FAIL/postdone. These are NOT the "old apply-before-commit ordering" tests task 7.3 owns (a different axis — ordering vs wire-format) — they are a DIRECT, in-scope, mechanical consequence of the wire-format fix itself, so I fixed them here rather than deferring: extended the helper to a `_ex` variant returning `_pad` too (existing helper becomes a thin wrapper, zero other call sites touched — same pattern as persist_submit_job_fact_ex), changed the wire-level assertions to check the new `_pad` marker, and added a direct call to the guard core (`apply_job_fail_transition`/`apply_job_postdone_transition`) on a scratch `job_t` to keep pinning the actual merge semantics the test was written to protect. A 3rd test (`test_fact_state_unchanged_throughout_quorum_loss_outage`) had the identical `st_upd` bit-check for postdone; fixed the same way (already had a `(void)st_fail` precedent in the same file for exactly this "order-dependent, can't check at receipt time" reasoning — extended it to `st_upd`/`pad_upd`). Lesson: when a task's fix legitimately changes what a durable record contains, grep for every DIRECT byte/field-level test assertion on that record (not just behavioral tests) before declaring the regression sweep clean — `grep cbrp_count_jobstate_entries` found them, `--run test_apply_pump` alone would not have.
|
||||
- (2026-07-04, same task) RED-phase verification for a bugfix already merged into the same commit as its own regression test (no separate "before" commit to check out) — reverted the exact 2 production hunks in-place (record_job_fail_fact's scratch-copy computation + apply_committed_entry's WAL_JOB_FAIL case ignoring `_pad`) using Edit, confirmed the new test fails with the EXACT predicted corrupted value (`Expected 264 Was 262` = DONE|PERR vs RUN|EXIT|PERR, byte-for-byte matching the hand-derived bug trace), then restored from a pre-saved `cp` snapshot of the fixed files and `diff`-verified byte-identical restoration before rebuilding GREEN. Combined with the existing longjmp-past-teardown lesson (isolate the new test alone in RUN_TEST via `#if 0` during the revert-and-check step, restore full list after) — both techniques compose cleanly.
|
||||
|
|
@ -423,3 +427,15 @@ Caught immediately by the build (gcc: "macro requires 3 arguments, but only 2 gi
|
|||
|
||||
## Key Learning (2026-07-05, bug-1416, raft-apply-decoupling task 7.2) — cluster_init_phase2's persist_recover() apply-correctness verification can false-positive APPLY_DIVERGENT when a returning node's recovery vtask races a compressed catch-up replication burst — FIXED round 2
|
||||
Found live, reproduced 10/10 times, building the `--factstream` kill-mid-fact-stream HA soak scenario. `cluster_init_phase2()` (cluster.c) calls `cluster_open_listener()` (accepts network I/O, including replicated/catch-up WAL writes) BEFORE `persist_recover()` (the task-5.1 boot-time apply-correctness verification, Req 8.1/8.2) — and `persist_recover()` itself fires from a vtask, not synchronously. `persist_verify_apply_correctness()`'s LIVE reads of `p->last_applied_lsn` and `count_jobmgr_entries()` can race two live mutation paths (the apply pump, `persist_apply_pump_run`/`persist_apply_pump_kick`; and the standby live-replication apply path, `ha_apply_single_entry`/`ha_apply_term_entry_via_library` in ha.c) that write those SAME values concurrently, producing a DIVERGENT verdict on a node that has NOT actually diverged. **FIXED (round 2, review remediation)**: new `_Atomic bool persist_t.recovery_complete` gate (persist.h/persist.c). `persist_apply_pump_kick()` no-ops while closed; `ha_apply_single_entry`/`ha_apply_term_entry_via_library` skip ONLY the in-memory `apply_wal_entry()` call while closed (durability write + ACK + applied_lsn advance stay ungated — mirrors bug-1043's durability/materialization decoupling). `persist_recover()` closes it at entry, reopens it + issues one deliberate catch-up kick before returning. **Round-2-of-the-fix gap, found via live verification, not just code review**: this alone was NOT enough — a node stuck retrying its own election as a losing/timing-out candidate can receive and durably APPLY live-replicated entries via the ungated (`persist_init` default=true) path for tens of seconds BEFORE `cluster_init_phase2()` ever calls `persist_recover()` for the first time, reproducing the identical false-DIVERGENT symptom through an earlier entry point `persist_recover()`'s own bracket can't see. Closed with `persist_mark_recovery_pending(&c.persist)` in `myapp_run()` (myapp.c), called right after `persist_init()`/`persist_start()` succeed and BEFORE `ha_start_election_async()`/`cluster_run()` can process any network traffic — closing the gate from the true earliest safe moment. `persist_init()`'s default stays TRUE so none of this spec's ~20 unit-test fixtures that construct a `persist_t` directly (bypassing `myapp_run`) are affected. Verified: live 3-node `--factstream` soak (2 rounds, kill+restart both) now PASSes clean, zero `APPLY_DIVERGENT` across 5 `persist_recover()` calls; broader `controller_unit_test` sweep (23 groups) shows zero new failures; real-divergence detection re-confirmed intact via the 3 dedicated `test_apply_correctness_verification.c` divergence tests. **Do-not-repeat lesson**: when gating a live path behind a "recovery in progress" flag, the flag's OWN bracket around the recovery function itself is not sufficient if OTHER code paths (election retry, live replication) can mutate the same state BEFORE that function is ever first called — the gate must close at the EARLIEST point in the real process lifecycle traffic could possibly begin, not just around the specific function whose output it protects.
|
||||
|
||||
## Key Learning (2026-07-07, raft-apply-decoupling task 7.4 O(n^2) apply-pump fix, CLOSED) — segment-lookup cache on `apply_pump_read_entry` fixes the regression; validated live A/B in one environment
|
||||
Root cause: `apply_pump_read_entry` (persist.c) redid a full `opendir`/`readdir`/segment-sort + a `pread` scan from byte 0 on EVERY single committed-LSN lookup; its only caller (`persist_apply_pump_run`) calls it once per LSN in strict ascending order with no gaps in the common case, making N contiguous applies O(N^2). Fix: 4 new `persist_t` fields (`apply_pump_cache_fd/seg_lsn/off/next_lsn`, guarded ENTIRELY by `apply_pump_lock`, never `p->lock`) cache the open segment fd + byte offset positioned for the next expected LSN; a fast path does a direct `pread` at the cached offset (hit → O(1), advance cache; miss → close+invalidate, fall through to the UNCHANGED slow path, which also now populates the cache on its own success branch before `fd` gets closed). Invalidated in `persist_init`/`persist_fini`/`persist_recover` (stale pre-recovery cursor) and `persist_truncate_wal_above` (segment could be ftruncate'd/unlinked away) — correctness never depends on the cache being right, only performance does (every miss path is the pre-existing, already-correct slow path). Live A/B in the SAME environment (single-node, release build, `git stash`/`pop` on persist.c/.h to swap fixed vs pre-fix): pre-fix N=2000→1.8s (1111/s), N=4000→13.3s (301/s) = **7.4x elapsed for 2x volume** (confirms superlinear); fixed N=2000→0.52s (3831/s), N=4000→0.76s (5249/s) = **1.46x elapsed for 2x volume**, throughput actually IMPROVING with N (fixed-cost amortization) — clean confirmation of the fix with zero superlinear scaling.
|
||||
|
||||
## Do-Not-Repeat (2026-07-07) — the synchronous `jobsub` (COMMIT_MODE_WAIT) request/reply path has a real, reproducible intermittent hang; NOT this task's bug, has its own out-of-scope status
|
||||
While benchmarking task 7.4's throughput fix, a single-connection burst of sequential `jobsub` REPL calls hung after a handful to a few hundred calls (variable: 7, 150, 778, 1579 — never a fixed threshold), with the SERVER-side commit completing near-instantly (visible in the controller log) but the CLIENT's `cbuf_get` on that connection never unblocking; a BRAND NEW connection to the same server succeeds immediately, so the server itself is not deadlocked. Ruled out REQ_BLOCK admission-cap exhaustion as the cause (raising `--vcpus` 4→32, i.e. cap 3→31, did not help — still hung after 13 calls). This is `jobsub`-specific (COMMIT_MODE_WAIT, synchronous reply-wait); `jfinish`/`jstart` (COMMIT_MODE_RETAIN, fire-and-forget `xce_notify`, no reply wait) never exhibited it even at N=4000 in one burst. Confirmed pre-existing/environmental, NOT caused by the apply-pump fix (reproduces identically on stashed pre-fix binaries). **Workaround, not a fix**: submit jobs in small waves (~20) over FRESH connections with generous per-wave retry — a hung wave doesn't poison later ones, and patient retrying eventually gets 100% through. **Do-not-repeat**: don't waste time hypothesis-testing admission-pool/vcpus tuning for this class of hang without first confirming server-side liveness via an independent fresh-connection probe — that single check (does a NEW connection succeed right now?) instantly distinguishes "server deadlocked" from "this one connection's reply got lost," which have completely different fixes/workarounds.
|
||||
|
||||
## Do-Not-Repeat (2026-07-07) — `$EPOCHREALTIME` read inside a polling loop with per-iteration subprocess calls can yield a wildly-wrong before/after diff; capture it as a single simple pair, not inline in deadline arithmetic
|
||||
A benchmark script computed elapsed time via `$EPOCHREALTIME` timestamps read inside a `while` polling loop that also computed its own timeout deadline via `awk` on every iteration — it printed a "commit-poll timeout after 180s" message in the SAME run whose reported `commit_elapsed` was 0.176s, an impossible combination. `$EPOCHREALTIME` itself is fine in isolation (verified: bash 5.3, two consecutive reads show a proper sub-ms-resolution increment) — the bug was in the surrounding script structure (mixing a `done_count`-vs-target break condition, a separate deadline-exceeded break condition, and per-iteration subprocess/awk calls all touching timing state), not the variable. **Fix that worked**: use bash's plain integer `SECONDS` builtin (`SECONDS=0` before, read `$SECONDS` after) for the loop's OWN deadline/elapsed bookkeeping, and take a SEPARATE single `t_start`/`t_end` `$EPOCHREALTIME` pair (captured once, diffed once via one final `awk` call) purely for sub-second-precision reporting. **Do-not-repeat**: never trust a timing result that contradicts an adjacent diagnostic message (e.g. a "timeout after Xs" alongside an elapsed value ≪X) — that contradiction itself is the tell that the measurement code (not the system under test) is broken; re-derive elapsed time via the simplest possible mechanism before trusting the number.
|
||||
|
||||
## Do-Not-Repeat (2026-07-07) — a joblist/committed-count poll loop must scope its match to THIS run's own job IDs, never a cluster-wide state count
|
||||
A benchmark script counted `done_count` as "every joblist row with state DONE" — on a controller that had accumulated DONE jobs from a PRIOR run in the same process lifetime, this produced an instant false-positive "already done" (0s elapsed) the moment the pre-existing DONE count alone exceeded the new batch's N, even though none of the NEW batch's facts had been sent yet. Fixed by writing the batch's own job IDs to a file and using `awk`'s `getline` to build a lookup set, counting only `($1 in ids) && $4=="DONE"`. **Do-not-repeat**: any poll-for-completion loop against a stateful, non-reset-between-runs server must scope its match to the specific IDs/records THIS run created — a bare aggregate count silently double-counts history from earlier runs against the same live process.
|
||||
|
|
|
|||
|
|
@ -1,470 +1,188 @@
|
|||
{
|
||||
"session_id": "session-2026-07-05-2050",
|
||||
"started": "2026-07-05T03:50:02.712Z",
|
||||
"session_id": "session-2026-07-09-0142",
|
||||
"started": "2026-07-09T08:42:25.044Z",
|
||||
"files_read": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/requirements.md": {
|
||||
"count": 3,
|
||||
"tokens": 4629,
|
||||
"first_read": "2026-07-05T05:13:49.214Z"
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/typemap.h": {
|
||||
"count": 2,
|
||||
"tokens": 5116,
|
||||
"first_read": "2026-07-09T09:03:21.212Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md": {
|
||||
"count": 12,
|
||||
"tokens": 20719,
|
||||
"first_read": "2026-07-05T05:14:01.912Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/design.md": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/schema/controller.capnp": {
|
||||
"count": 1,
|
||||
"tokens": 9210,
|
||||
"first_read": "2026-07-05T05:14:02.728Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c": {
|
||||
"count": 4,
|
||||
"tokens": 28490,
|
||||
"first_read": "2026-07-05T05:14:31.226Z"
|
||||
"tokens": 9720,
|
||||
"first_read": "2026-07-09T09:04:23.282Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft.h": {
|
||||
"count": 1,
|
||||
"tokens": 17581,
|
||||
"first_read": "2026-07-05T05:15:27.928Z"
|
||||
"tokens": 17836,
|
||||
"first_read": "2026-07-09T09:04:52.309Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test.c": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T05:20:15.107Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md": {
|
||||
"count": 4,
|
||||
"tokens": 4553,
|
||||
"first_read": "2026-07-05T05:28:34.593Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh": {
|
||||
"count": 18,
|
||||
"tokens": 18920,
|
||||
"first_read": "2026-07-05T05:28:47.075Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c": {
|
||||
"count": 12,
|
||||
"tokens": 29726,
|
||||
"first_read": "2026-07-05T05:29:07.259Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/job.c": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c": {
|
||||
"count": 5,
|
||||
"tokens": 16424,
|
||||
"first_read": "2026-07-05T05:30:09.491Z"
|
||||
"tokens": 103039,
|
||||
"first_read": "2026-07-09T09:05:08.232Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client.c": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md": {
|
||||
"count": 6,
|
||||
"tokens": 721,
|
||||
"first_read": "2026-07-09T09:10:19.727Z"
|
||||
},
|
||||
"/home/claude/.claude/projects/-home-rshen-projects-xce-worktrees-raft-tla/5e55cf26-d05d-4a2e-8196-d2c70e0c063c/tool-results/bewx6l1jq.txt": {
|
||||
"count": 2,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T05:30:41.791Z"
|
||||
"first_read": "2026-07-09T09:40:56.341Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/cluster.c": {
|
||||
"count": 4,
|
||||
"tokens": 18098,
|
||||
"first_read": "2026-07-05T05:30:54.661Z"
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-deploy-ha/SKILL.md": {
|
||||
"count": 1,
|
||||
"tokens": 1351,
|
||||
"first_read": "2026-07-09T09:42:32.594Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md": {
|
||||
"count": 3,
|
||||
"tokens": 4818,
|
||||
"first_read": "2026-07-09T09:49:08.353Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/xce/errnox.h": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-09T10:15:10.471Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client_internal.h": {
|
||||
"count": 2,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T05:31:49.690Z"
|
||||
"first_read": "2026-07-09T10:21:05.195Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/sched.c": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T05:32:31.884Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c": {
|
||||
"count": 6,
|
||||
"tokens": 5404,
|
||||
"first_read": "2026-07-05T05:33:41.172Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/wal_types.h": {
|
||||
"count": 2,
|
||||
"tokens": 1996,
|
||||
"first_read": "2026-07-05T05:33:51.925Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/xce/types.h": {
|
||||
"count": 3,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T05:34:30.060Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/soak-verdict.sh": {
|
||||
"count": 3,
|
||||
"tokens": 1383,
|
||||
"first_read": "2026-07-05T05:37:16.616Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c": {
|
||||
"count": 20,
|
||||
"tokens": 60786,
|
||||
"first_read": "2026-07-05T06:14:23.860Z"
|
||||
},
|
||||
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/full.diff": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T06:47:57.965Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/utils.c": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T06:51:04.731Z"
|
||||
},
|
||||
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/soak.diff": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T06:59:21.281Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c": {
|
||||
"count": 9,
|
||||
"tokens": 102306,
|
||||
"first_read": "2026-07-05T07:06:14.209Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h": {
|
||||
"count": 5,
|
||||
"tokens": 19990,
|
||||
"first_read": "2026-07-05T07:09:10.020Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/xcectx.h": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T07:12:14.838Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/vtaskmgr.c": {
|
||||
"count": 1,
|
||||
"tokens": 3822,
|
||||
"first_read": "2026-07-05T07:19:54.733Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/controller_test.c": {
|
||||
"count": 2,
|
||||
"tokens": 1986,
|
||||
"first_read": "2026-07-05T07:25:28.359Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/myapp.c": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client_failover.c": {
|
||||
"count": 4,
|
||||
"tokens": 5215,
|
||||
"first_read": "2026-07-05T07:31:53.169Z"
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-09T10:21:22.224Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_apply_correctness_verification.c": {
|
||||
"count": 4,
|
||||
"tokens": 17000,
|
||||
"first_read": "2026-07-05T08:08:11.151Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/cluster.c": {
|
||||
"count": 1,
|
||||
"tokens": 380,
|
||||
"first_read": "2026-07-05T08:09:36.337Z"
|
||||
"tokens": 18444,
|
||||
"first_read": "2026-07-09T10:24:23.282Z"
|
||||
},
|
||||
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/tacv_out.log": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c": {
|
||||
"count": 1,
|
||||
"tokens": 30294,
|
||||
"first_read": "2026-07-09T10:25:37.035Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client.c": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-05T12:13:07.549Z"
|
||||
"first_read": "2026-07-09T10:25:53.187Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh": {
|
||||
"count": 1,
|
||||
"tokens": 19391,
|
||||
"first_read": "2026-07-09T10:29:23.532Z"
|
||||
}
|
||||
},
|
||||
"files_written": [
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/reports/jun29-jul04-2026.md",
|
||||
"action": "create",
|
||||
"tokens": 979,
|
||||
"at": "2026-07-05T03:57:01.756Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 40,
|
||||
"at": "2026-07-05T05:20:38.611Z"
|
||||
"tokens": 30,
|
||||
"at": "2026-07-09T08:58:09.645Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/schema/controller.capnp",
|
||||
"action": "edit",
|
||||
"tokens": 38,
|
||||
"at": "2026-07-05T05:20:58.465Z"
|
||||
"tokens": 31,
|
||||
"at": "2026-07-09T09:04:29.441Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/typemap.h",
|
||||
"action": "edit",
|
||||
"tokens": 26,
|
||||
"at": "2026-07-05T05:21:03.400Z"
|
||||
"tokens": 22,
|
||||
"at": "2026-07-09T09:04:43.146Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft.h",
|
||||
"action": "edit",
|
||||
"tokens": 25,
|
||||
"at": "2026-07-05T05:21:17.748Z"
|
||||
"tokens": 22,
|
||||
"at": "2026-07-09T09:04:58.815Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
|
||||
"action": "edit",
|
||||
"tokens": 25,
|
||||
"at": "2026-07-05T05:25:03.542Z"
|
||||
"tokens": 23,
|
||||
"at": "2026-07-09T09:05:17.167Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
|
||||
"action": "edit",
|
||||
"tokens": 896,
|
||||
"at": "2026-07-05T05:25:21.590Z"
|
||||
"tokens": 21,
|
||||
"at": "2026-07-09T09:05:29.515Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 46,
|
||||
"at": "2026-07-05T05:41:04.648Z"
|
||||
"tokens": 328,
|
||||
"at": "2026-07-09T09:11:54.033Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 97,
|
||||
"at": "2026-07-05T05:41:12.545Z"
|
||||
"tokens": 90,
|
||||
"at": "2026-07-09T09:14:56.189Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 825,
|
||||
"at": "2026-07-05T05:41:30.455Z"
|
||||
"tokens": 336,
|
||||
"at": "2026-07-09T09:19:57.240Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
|
||||
"action": "edit",
|
||||
"tokens": 566,
|
||||
"at": "2026-07-05T05:41:44.220Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c",
|
||||
"action": "edit",
|
||||
"tokens": 415,
|
||||
"at": "2026-07-05T05:42:04.666Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c",
|
||||
"action": "edit",
|
||||
"tokens": 376,
|
||||
"at": "2026-07-05T05:42:14.873Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/soak-verdict.sh",
|
||||
"action": "edit",
|
||||
"tokens": 363,
|
||||
"at": "2026-07-05T05:42:57.027Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 450,
|
||||
"at": "2026-07-05T05:44:13.870Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 344,
|
||||
"at": "2026-07-05T05:44:41.636Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 54,
|
||||
"at": "2026-07-05T05:44:48.489Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 166,
|
||||
"at": "2026-07-05T05:45:10.893Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 2683,
|
||||
"at": "2026-07-05T05:46:02.987Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 568,
|
||||
"at": "2026-07-05T05:46:28.157Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 871,
|
||||
"at": "2026-07-05T05:48:15.351Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 228,
|
||||
"at": "2026-07-05T05:48:37.033Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 776,
|
||||
"at": "2026-07-05T05:53:15.669Z"
|
||||
"at": "2026-07-09T09:33:24.688Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 17,
|
||||
"at": "2026-07-05T05:53:37.328Z"
|
||||
"tokens": 385,
|
||||
"at": "2026-07-09T09:34:18.859Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 318,
|
||||
"at": "2026-07-05T05:53:47.430Z"
|
||||
"tokens": 386,
|
||||
"at": "2026-07-09T09:41:11.513Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 141,
|
||||
"at": "2026-07-05T05:54:00.332Z"
|
||||
"tokens": 510,
|
||||
"at": "2026-07-09T09:50:04.243Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-deploy-ha/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 490,
|
||||
"at": "2026-07-09T09:50:23.179Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 374,
|
||||
"at": "2026-07-05T06:16:57.614Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 613,
|
||||
"at": "2026-07-05T06:22:34.527Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 13,
|
||||
"at": "2026-07-05T06:23:51.836Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 210,
|
||||
"at": "2026-07-05T06:24:37.656Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
|
||||
"action": "edit",
|
||||
"tokens": 92,
|
||||
"at": "2026-07-05T06:24:53.835Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
|
||||
"action": "edit",
|
||||
"tokens": 245,
|
||||
"at": "2026-07-05T06:33:11.363Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 603,
|
||||
"at": "2026-07-05T06:34:12.298Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 160,
|
||||
"at": "2026-07-05T06:34:36.760Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 993,
|
||||
"at": "2026-07-05T07:02:41.181Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
|
||||
"action": "edit",
|
||||
"tokens": 680,
|
||||
"at": "2026-07-05T07:16:17.567Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
|
||||
"action": "edit",
|
||||
"tokens": 292,
|
||||
"at": "2026-07-05T07:16:28.893Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
|
||||
"action": "edit",
|
||||
"tokens": 160,
|
||||
"at": "2026-07-05T07:16:43.698Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
|
||||
"action": "edit",
|
||||
"tokens": 133,
|
||||
"at": "2026-07-05T07:16:50.545Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
|
||||
"action": "edit",
|
||||
"tokens": 514,
|
||||
"at": "2026-07-05T07:17:19.666Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
|
||||
"action": "edit",
|
||||
"tokens": 362,
|
||||
"at": "2026-07-05T07:19:06.213Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
|
||||
"action": "edit",
|
||||
"tokens": 403,
|
||||
"at": "2026-07-05T07:19:17.398Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
|
||||
"action": "edit",
|
||||
"tokens": 313,
|
||||
"at": "2026-07-05T07:21:05.529Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
|
||||
"action": "edit",
|
||||
"tokens": 504,
|
||||
"at": "2026-07-05T07:21:38.054Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
|
||||
"action": "edit",
|
||||
"tokens": 311,
|
||||
"at": "2026-07-05T07:21:56.305Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
|
||||
"action": "edit",
|
||||
"tokens": 486,
|
||||
"at": "2026-07-05T07:39:27.113Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
|
||||
"action": "edit",
|
||||
"tokens": 164,
|
||||
"at": "2026-07-05T07:39:49.040Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/myapp.c",
|
||||
"action": "edit",
|
||||
"tokens": 384,
|
||||
"at": "2026-07-05T07:39:59.983Z"
|
||||
"tokens": 583,
|
||||
"at": "2026-07-09T10:29:48.034Z"
|
||||
}
|
||||
],
|
||||
"edit_counts": {
|
||||
"reports/jun29-jul04-2026.md": 1,
|
||||
"src/lib/raft/raft_sim.c": 4,
|
||||
".kiro/specs/raft-apply-decoupling/tasks.md": 2,
|
||||
"src/controller/testsuite/test.c": 5,
|
||||
"src/controller/testsuite/waldump.c": 2,
|
||||
".claude/skills/xce-test-ha/soak-verdict.sh": 1,
|
||||
".claude/skills/xce-test-ha/run-ha-soak.sh": 16,
|
||||
".claude/skills/xce-test-ha/SKILL.md": 4,
|
||||
"src/include/persist.h": 3,
|
||||
"src/controller/persist.c": 7,
|
||||
".claude/skills/xce-build/SKILL.md": 7,
|
||||
"src/schema/controller.capnp": 1,
|
||||
"src/include/typemap.h": 1,
|
||||
"src/lib/raft/raft.h": 1,
|
||||
"src/controller/ha.c": 2,
|
||||
"src/controller/myapp.c": 1
|
||||
".claude/skills/xce-test-ha/SKILL.md": 1,
|
||||
".claude/skills/xce-deploy-ha/SKILL.md": 1,
|
||||
".claude/skills/xce-test-ha/run-ha-soak.sh": 1
|
||||
},
|
||||
"anatomy_hits": 20,
|
||||
"anatomy_misses": 11,
|
||||
"repeated_reads_warned": 104,
|
||||
"anatomy_hits": 10,
|
||||
"anatomy_misses": 5,
|
||||
"repeated_reads_warned": 17,
|
||||
"cerebrum_warnings": 0,
|
||||
"stop_count": 8
|
||||
"stop_count": 33
|
||||
}
|
||||
|
|
@ -4140,3 +4140,4 @@
|
|||
| 04:22 | bug-1422 fixed: verify_no_loss false-positive from client retry-storm race post-election; added bounded 3x retry; 40-rnd seq+chaotic both PASS (123 jobs, zero loss) | .claude/skills/xce-test-ha/run-ha-soak.sh,buglog.json | pass | ~3k |
|
||||
| 04:22 | Session end: 15 writes across 6 files (SKILL.md, controller.capnp, typemap.h, raft.h, ha.c) | 15 reads | ~214763 tok |
|
||||
| 04:22 | Session end: 15 writes across 6 files (SKILL.md, controller.capnp, typemap.h, raft.h, ha.c) | 15 reads | ~214763 tok |
|
||||
| 04:57 | Session end: 15 writes across 6 files (SKILL.md, controller.capnp, typemap.h, raft.h, ha.c) | 15 reads | ~214763 tok |
|
||||
|
|
|
|||
29047
.wolf/token-ledger.json
29047
.wolf/token-ledger.json
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue