diff --git a/.claude/skills/gitea-sync/SKILL.md b/.claude/skills/gitea-sync/SKILL.md index b9dc26f3..c74e5d32 100644 --- a/.claude/skills/gitea-sync/SKILL.md +++ b/.claude/skills/gitea-sync/SKILL.md @@ -1,7 +1,7 @@ --- name: gitea-sync description: Sync cc-sdd specs and OpenWolf buglog to Gitea milestones/issues in nexa/xce -argument-hint: [--dry-run] [--spec ] [--bugs] [--since ] +argument-hint: [--dry-run] [--spec ] [--bugs] [--since ] [--ids ] triggers: - /gitea-sync - sync to gitea @@ -50,6 +50,11 @@ Keep the script and this procedure in step when changing either. date; for already-synced bugs (`gitea_issue` set), also mirror those whose `status_updated >=` this date, so a status change on an old bug still reaches Gitea without flooding it with old never-synced bugs +- `--ids `: also sync these bugs regardless of dates (OR'd + with `--since`). Needed because bulk triage has bumped `last_seen` / + `status_updated` on most of the buglog, and for a never-synced bug + `--since` only checks `last_seen` — a recently fixed older bug is otherwise + unreachable. Always `--dry-run` first: `--since` alone can match ~1100 bugs. ### 1. Ensure labels exist diff --git a/.wolf/STATUS.md b/.wolf/STATUS.md index b396bfc8..3e24b443 100644 --- a/.wolf/STATUS.md +++ b/.wolf/STATUS.md @@ -7,20 +7,20 @@ ## ✅ Done -- **scheduler-sim-eager-reconciliation spec, revision 4** (2026-09-12, commit `25fb5c11`): captured jobs' `job->state` now set to `JOB_STAT_RUN` synchronously at dispatch capture (`sim_reconcile_unaccounted_jobs`), mirroring normal-mode's `sched_commit_decision` semantics (bug-1624/bug-1804). All 4 revision tasks in `tasks.md` checked off. -- **bug-1799/1801**: fixed sim replay resInUse over-commit — `sched_account_jobs`'s per-round re-derive now skips RECONCILING/UNAVAIL workers (matches `core_reset_workers`'s existing bug-1751 skip). Previously a worker's `used` slots could exceed its own `value` capacity after a resync replay (observed 26 used vs. 22 capacity). -- **bug-1800**: bytes-valued CONSUMABLE resources (mem/swp/tmp) in `info worker` now render with human-scaled units, matching maxmem/maxswap/maxtmp's existing formatting. -- **bug-1802/1803**: `break ANY` breakpoint kind added then reverted after live feedback it over-fired inside busy sessions; replaced with a prescan-based "next non-dummy session" hint surfaced in `info summary`. -- Full scheduler test suite verified GREEN: 77/77 tests pass (asan profile, `xce.build`), confirmed via fresh build+test run before committing. +- **bug-1754** (2026-09-26, `6876e4f6`, pushed): a failed job's exit code is now persisted. A worker reports every nonzero exit as a FAIL (not FINISH), and WAL_JOB_FAIL had no code slot, so `job.exit_info` was NULL for every failed job. WAL_JOB_FAIL now carries an int32 code trailer (`persist_submit_job_fail_fact`); both apply seams adopt it via `apply_job_fail_exit_code()` (non-POST_EXEC stages only). `apply_wal_entry` also no longer assigns wire state 0 for STAGE_FLAG FAIL entries. New `test_wal_job_fail_fact.c`; controller suite 1228/0. Soaks: sequential 6 rounds PASS (SIMVERIFY 7 files, 0 mismatches) + factstream 2 rounds PASS. +- **HA stability fixes** (2026-09-25, pushed, soak-verified): bug-1864 (per-endpoint FIFO in vtask dispatcher), bug-1865 (primary control endpoint refcounted, no double-close), bug-1850 (NOT_READY retries in place, no failover rotate), bug-1854 (re-pushed resource defs are idempotent). +- **bug-1714 / bug-1840** marked cannot-reproduce (0 crashes across 2,400-element burst on old + current builds). +- **Gitea sync** (2026-09-26): issues #264-#286 created (recent bugs + bug-1798..1804, 1714, 1754, 1850, 1854), #74 closed. Only a subset of the buglog is mirrored by design; `scripts/gitea-sync.py` gained `--ids` because bulk triage bumped the date fields `--since` relies on. +- Logged: **bug-1866** (worker `launch_type` leak), **bug-1867** (controller snapshot drops `exit_info` — capnp `Job` has no exit field). --- ## 🚀 Next phase -**Goal:** _None currently active — last quest (scheduler-sim-eager-reconciliation revision + bug-1799..1804) is committed and closed out._ +**Goal:** _None active._ Suggested order: bug-1508 (fetchfile path check, security) → bug-1747 FAIL half (add finish_time to the new WAL_JOB_FAIL trailer) → bug-1867 → `xce_test` harness batch + bug-1866 → a tasks-ready spec. ### Open items (not yet actioned) -- `gitea` MCP server failed to connect this session (CONNECT_TIMEOUT) — buglog entries bug-1798 through bug-1804 have not been synced to Gitea issues via the `gitea-sync` skill; retry once the server is reachable. +- _none pending_ ### Files to create / edit - _none pending_ @@ -33,16 +33,21 @@ ## 📁 Active architecture - **Stack:** Go API/CLI tier (`src/frontend/apiserver`) fronting C daemons (`src/controller`, `src/worker`, `src/scheduler`); Meson-based build via `src/tools/xce-build.sh`, hermetic nix dev shell. -- **Current focus area:** `src/scheduler` — sim/replay mode (`sim.c`, `sim_loop.c`, `repl_info.c`) and resource accounting (`resource.c`). +- **Current focus area:** `src/controller` persistence — WAL fact payloads and apply seams (`persist.c` `apply_committed_entry` / `apply_wal_entry`, `job.c` `record_job_*_fact`). --- ## 🔧 Useful commands ```bash -# Build + test scheduler only (ASAN profile) -bash src/tools/xce-build.sh scheduler asan xce.build --jobs +# Build controller unit tests in place (nix shell) and run one group +cd src && nix develop . --command meson compile -C xce.build controller_unit_test xce_controller +ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test -t test_wal_job_fail_fact -# Replay a live-captured scheduler session -./xce.build/scheduler/xce_scheduler -i -s +# HA soaks (sequential with real worker+scheduler; worker fact stream) +XCE_HA_ROUNDS=6 nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh --sequential +nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh --factstream + +# Gitea bug sync (REST, no MCP needed) — always dry-run first +python3 scripts/gitea-sync.py --dry-run --bugs --since [--ids bug-N,bug-M] ``` diff --git a/.wolf/buglog.json b/.wolf/buglog.json index da89ba67..9cdc059c 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -15785,6 +15785,8 @@ }, { "id": "bug-1429", + "gitea_synced_status": "fixed", + "gitea_issue": 264, "status_note": "b5d59253", "status": "fixed", "status_updated": "2026-09-25", @@ -23648,6 +23650,8 @@ }, { "id": "bug-1714", + "gitea_synced_status": "cannot-reproduce", + "gitea_issue": 265, "repro_attempts": "0/2 runs, 2026-09-25: documented recipe (3-node HA + worker + scheduler, 8 waves x 6 concurrent 'jobsub -a 50' = 2400 array elements each) on b63bb581 build.dbg and on d87b78dc-era ASAN build -- no crash, scheduler alive throughout, 0 redelivered array pushes (0 'sched_add_job failed' duplicates; the original repro saw up to 18x redelivery, the recipe's precondition)", "status": "cannot-reproduce", "status_updated": "2026-09-25", @@ -24850,6 +24854,8 @@ }, { "id": "bug-1754", + "gitea_synced_status": "fixed", + "gitea_issue": 266, "status": "fixed", "status_updated": "2026-09-26", "status_note": "6876e4f6: WAL_JOB_FAIL carries an int32 code trailer; both apply seams adopt it into exit_info (non-POST_EXEC stages). Also fixed apply_wal_entry assigning wire state 0 for STAGE_FLAG FAIL entries.", @@ -26071,6 +26077,8 @@ }, { "id": "bug-1798", + "gitea_synced_status": "fixed", + "gitea_issue": 267, "status": "fixed", "status_updated": "2026-09-24", "timestamp": "2026-09-10T06:57:00Z", @@ -26099,6 +26107,8 @@ }, { "id": "bug-1799", + "gitea_synced_status": "fixed", + "gitea_issue": 268, "status": "fixed", "status_updated": "2026-09-24", "status_note": "25fb5c11", @@ -26140,6 +26150,8 @@ }, { "id": "bug-1800", + "gitea_synced_status": "fixed", + "gitea_issue": 269, "status": "fixed", "status_updated": "2026-09-24", "timestamp": "2026-09-11T07:55:00Z", @@ -26174,6 +26186,8 @@ }, { "id": "bug-1801", + "gitea_synced_status": "fixed", + "gitea_issue": 270, "status": "fixed", "status_updated": "2026-09-24", "timestamp": "2026-09-11T08:35:00Z", @@ -26210,6 +26224,8 @@ }, { "id": "bug-1802", + "gitea_synced_status": "fixed", + "gitea_issue": 271, "status": "fixed", "status_updated": "2026-09-24", "timestamp": "2026-09-11T08:35:00Z", @@ -26238,6 +26254,8 @@ }, { "id": "bug-1803", + "gitea_synced_status": "fixed", + "gitea_issue": 272, "status": "fixed", "status_updated": "2026-09-24", "timestamp": "2026-09-11T09:35:00Z", @@ -26269,6 +26287,8 @@ }, { "id": "bug-1804", + "gitea_synced_status": "fixed", + "gitea_issue": 273, "status": "fixed", "status_updated": "2026-09-24", "timestamp": "2026-09-11T09:55:00Z", @@ -27404,7 +27424,7 @@ { "id": "bug-1840", "repro_attempts": "0/1 ASAN run, 2026-09-25: 3-node HA + worker + scheduler, 8 waves x 6 concurrent 'jobsub -a 50' (2400 elements) on the current ASAN build -- no AddressSanitizer report from the scheduler; plus ~10 ASAN HA soaks today (sequential/chaotic, all with scheduler replay-verify) with no scheduler heap error", - "gitea_synced_status": "investigating", + "gitea_synced_status": "cannot-reproduce", "gitea_issue": 74, "status": "cannot-reproduce", "status_updated": "2026-09-25", @@ -27691,6 +27711,8 @@ }, { "id": "bug-1850", + "gitea_synced_status": "fixed", + "gitea_issue": 274, "status": "fixed", "status_updated": "2026-09-25", "status_note": "6b1862f8; soaks 2026-09-25: 0 NOT_READY-triggered rotations, 1 live same-controller retry (worker, recovered in 1.2s), all PASS", @@ -27765,6 +27787,8 @@ }, { "id": "bug-1854", + "gitea_synced_status": "fixed", + "gitea_issue": 275, "status": "fixed", "status_updated": "2026-09-25", "status_note": "663ea63c", @@ -27829,6 +27853,8 @@ }, { "id": "bug-1857", + "gitea_synced_status": "fixed", + "gitea_issue": 276, "status": "fixed", "status_updated": "2026-09-24", "status_note": "3cc2f6e3", @@ -27849,6 +27875,8 @@ }, { "id": "bug-1858", + "gitea_synced_status": "fixed", + "gitea_issue": 277, "status": "fixed", "status_updated": "2026-09-24", "status_note": "9e9f2798", @@ -27870,6 +27898,8 @@ }, { "id": "bug-1859", + "gitea_synced_status": "fixed", + "gitea_issue": 278, "status": "fixed", "status_updated": "2026-09-25", "status_note": "7c98c056", @@ -27892,6 +27922,8 @@ }, { "id": "bug-1860", + "gitea_synced_status": "open", + "gitea_issue": 279, "status": "open", "status_updated": "2026-09-24", "status_note": "bug-1812 guard blocks authoritative PEND after disconnected requeue; needs per-job freshness in JOBS_PUSH", @@ -27914,6 +27946,8 @@ }, { "id": "bug-1861", + "gitea_synced_status": "fixed", + "gitea_issue": 280, "status": "fixed", "status_updated": "2026-09-25", "status_note": "b5d59253", @@ -27934,6 +27968,8 @@ }, { "id": "bug-1862", + "gitea_synced_status": "fixed", + "gitea_issue": 281, "status": "fixed", "status_updated": "2026-09-25", "status_note": "c29518b2", @@ -27956,6 +27992,8 @@ }, { "id": "bug-1863", + "gitea_synced_status": "fixed", + "gitea_issue": 282, "status": "fixed", "status_updated": "2026-09-25", "status_note": "aea34738", @@ -27976,6 +28014,8 @@ }, { "id": "bug-1864", + "gitea_synced_status": "fixed", + "gitea_issue": 283, "status": "fixed", "status_updated": "2026-09-25", "status_note": "0d3bb965", @@ -27999,6 +28039,8 @@ }, { "id": "bug-1865", + "gitea_synced_status": "fixed", + "gitea_issue": 284, "status": "fixed", "status_updated": "2026-09-25", "status_note": "910b43cd; soaks 2026-09-25 (2 chaotic + 1 sequential) fdatasync_failed=0, all PASS", @@ -28022,6 +28064,8 @@ }, { "id": "bug-1866", + "gitea_synced_status": "open", + "gitea_issue": 285, "status": "open", "status_updated": "2026-09-25", "status_note": "remove_worker never frees worker->launch_type", @@ -28041,6 +28085,8 @@ }, { "id": "bug-1867", + "gitea_synced_status": "open", + "gitea_issue": 286, "status": "open", "status_updated": "2026-09-26", "status_note": "found while fixing bug-1754; needs a Job.exitCode capnp field + encode/decode, snapshot round-trip test", diff --git a/.wolf/cerebrum.md b/.wolf/cerebrum.md index f3771570..fda6800d 100644 --- a/.wolf/cerebrum.md +++ b/.wolf/cerebrum.md @@ -151,6 +151,7 @@ - **(2026-08-26, bug-1713) In this sandbox's local-loopback 3-node deploy, the FIRST-BOOT election-settling window varies wildly run to run (sometimes near-instant, sometimes ~10-12s via a term-1 election-timeout-then-PreVote-failover fallback path) — a fixed "wait for start-all.sh's health-check, then fire N requests" tight loop is NOT a reliable way to catch a transient `E_XCE_NOT_READY`/`E_XCE_NOMASTER` window: by the time start-all.sh's own 1s-granularity health-check first observes `CTRL_PRIMARY` in a log and returns, `ha_has_write_quorum()`'s `standby_endpoints` (populated synchronously inside `raft_cb_become_leader` from the raft library's own connected-voter set) is typically ALREADY satisfied, so the gate is already open. A REAL FAILOVER (kill -9 the current primary, then hammer the survivors with NO sleep, no health-check wait) reproduced the exact live symptom reliably instead — got a genuine bare `-10013` at ~t+8.6s post-kill on an unfixed client, and confirmed zero such failures surfaced across the equivalent window on the fixed client. The bug's own root-cause writeup already named this as an equally-valid trigger ("...or any time a real failover is in flight") — prefer the failover trigger over racing first-boot timing when a live repro is needed and cold-boot timing proves flaky. - **(2026-08-26) A hand-rolled multi-connection mock TCP/TLS server in a test file needs a genuinely FINITE accept loop (`for handler in handlers:`), not `while self._handlers:` guarding a list that's never mutated — the latter is an always-true condition that leaves the server thread blocked in `accept()` indefinitely once the test's real client stops connecting, which does not fail the test but silently adds `thread.join(timeout=N)`'s FULL timeout to every such test's teardown (measured: 3 tests each added a flat 5s of pure teardown wait with zero effect on pass/fail).** Match the mock's handler-list length exactly to the number of connections the test actually expects (assert on `connection_count` for the "stops on its own budget" property instead of over-provisioning handlers "just in case") and additionally set a short `socket.settimeout()` on the listener as a safety net so a hypothetical future regression that causes MORE connections than expected still fails within one timeout window instead of silently reintroducing the same multi-second stall. - **[2026-09-24: open/closed now lives in each entry's `status` field (`.claude/rules/buglog-status.md`), not in `tags` or `fix` text.]** **(2026-09-19, `/xce-audit-invariants scheduler` after `scheduler-sim-live-allocation`) A buglog entry's `tags` array can say `"fixed"` while its own `fix` text says `"Not yet fixed"` — a self-contradiction that survives silently unless something actually re-reads the entry, because most tooling in this project checks tags, not the narrative field.** Found on bug-1682 (and its unnoticed duplicate, bug-1492 — same defect, `plugin_end_session()` guarding on `begin_session_fn` instead of `end_session_fn`, independently filed twice 7 weeks apart because bug-1681's investigation never cross-referenced the earlier entry) via an audit subagent that read BOTH the code and the buglog text, not just the tags — it caught that `plugin.c:441` was still visibly broken despite the tags claiming otherwise. **Lesson 1**: when closing out a large feature and bulk-stamping buglog entries as fixed/superseded, verify each stamped entry's `fix` narrative actually matches the tag before writing it — a tag-only update is a claim with no evidence behind it. **Lesson 2**: before filing a new bug for a defect discovered "while fixing X," grep buglog.json for the exact symptom text first (e.g. `grep -i "begin_session_fn.*end_session_fn"`) — this exact one-line guard/call mismatch was independently re-discovered and re-filed as a "new" bug once already in this project's history. +- **(2026-09-26, bug-1754) A worker reports a job that exits NONZERO as a FAIL fact, not a FINISH: `xcecd_job_finished()` (worker/src/controller_connector.c) calls `xcecd_notify_job_fail(jobid, stage, exit_code)` whenever `exit_code != 0`, and only sends `xcecli_notify_jobfinish` for exit 0.** So `on_job_finish`/WAL_JOB_FINISH only ever sees successful jobs in production, and anything about a failed job's exit code (exit_info, JOB_FAIL hook payloads, apiserver notify) lives on the `on_job_fail` -> `record_job_fail_fact` -> WAL_JOB_FAIL path, where `job_failinfo_t.code` is the job's real exit code for MAIN-stage failures (but the post-exec script's code for POST_EXEC/Q_POST_EXEC, and a -errno for controller-internal `mark_job_fail` callers). Check both paths before assuming a job-exit fix is complete. ## Do-Not-Repeat - **(2026-09-24) Never hand-edit or re-serialize `.wolf/buglog.json` to log a bug or change its status; use the `buglog` skill (`scripts/buglog.py add | status | note | recur | list | show | triage`).** Each entry's `status` (open / partial / investigating / fixed / cannot-reproduce / wontfix / duplicate, defined in `.claude/rules/buglog-status.md`) is the only source of truth for open/closed. Never infer it from `fix` text or `tags`; both drifted (bug-1772, bug-1682), and gitea-sync closed unfixed bugs by trusting a non-empty `fix` (bug-1855). `add` picks max id + 1. OpenWolf's hook uses entry count + 1, which collides, so its auto-detection is off (bug-1856). After a fix commit: `scripts/buglog.py status fixed --note `. diff --git a/scripts/gitea-sync.py b/scripts/gitea-sync.py index 6da433b0..f00cbdaa 100755 --- a/scripts/gitea-sync.py +++ b/scripts/gitea-sync.py @@ -788,7 +788,7 @@ def _resolve_label_ids(api, label_names): # ── Bug Sync ───────────────────────────────────────────────────────────────── -def sync_bugs(api, dry_run=False, quiet=False, since=None): +def sync_bugs(api, dry_run=False, quiet=False, since=None, ids=None): """Sync buglog entries to Gitea issues. Open/closed and the status label come from each entry's `status` (see @@ -801,6 +801,10 @@ def sync_bugs(api, dry_run=False, quiet=False, since=None): since: ISO date (YYYY-MM-DD). Create issues only for bugs with last_seen >= since; update already-synced ones whose last_seen or status_updated >= since. + ids: set of bug ids to sync regardless of dates. Combined with + since, a bug is synced if it matches either -- bulk triage + has bumped last_seen/status_updated across most of the log, + so dates alone cannot pick out recently fixed older bugs. """ if not BUGLOG_PATH.exists(): warn(f"buglog not found: {BUGLOG_PATH}") @@ -827,10 +831,14 @@ def sync_bugs(api, dry_run=False, quiet=False, since=None): # --since: new issues only for recently seen bugs; status changes # (status_updated) are still mirrored for already-synced ones. - if since: - recent = (bug.get("last_seen") or "") >= since - if bug.get("gitea_issue"): - recent = recent or (bug.get("status_updated") or "") >= since + if since or ids: + recent = False + if since: + recent = (bug.get("last_seen") or "") >= since + if bug.get("gitea_issue"): + recent = recent or (bug.get("status_updated") or "") >= since + if ids and bid in ids: + recent = True if not recent: skipped += 1 continue @@ -972,6 +980,8 @@ def main(): help="Sync bugs only") parser.add_argument("--since", metavar="DATE", help="Create issues only for bugs last seen >= DATE; update synced ones changed since (YYYY-MM-DD)") + parser.add_argument("--ids", metavar="ID[,ID...]", + help="Also sync these bug ids regardless of --since (comma-separated, e.g. bug-1798,bug-1799)") parser.add_argument("--quiet", action="store_true", help="Suppress progress output, print only errors") args = parser.parse_args() @@ -1007,7 +1017,10 @@ def main(): # Sync bugs if not args.spec: - sync_bugs(api, dry_run=args.dry_run, quiet=args.quiet, since=args.since) + ids = ({i.strip() for i in args.ids.split(",") if i.strip()} + if args.ids else None) + sync_bugs(api, dry_run=args.dry_run, quiet=args.quiet, + since=args.since, ids=ids) if not args.quiet: if args.dry_run: