fix(wolf): log 4 reproduced Gitea issues (#40-42, #44) + 1 discovered gitea-sync bug
Live-reproduced and root-caused on branch scheduler-framework via a single-node deploy: - bug-1673 (gitea #40): apiserver hosts-list MAX_SLOT stuck at 0 for workers not exact-name-matched in hosts.yaml -- display-only, C-side scheduling correctly uses the wildcard 'default' entry's slot count. Reporter's secondary "hard 8-job cap" claim did NOT reproduce. - bug-1674 (gitea #41): `xcecli jobs list -l` fails outright (not just missing timestamp fields as reported) -- two call sites validate field lists against JobSubmitRequest instead of JobInfo. - bug-1675 (gitea #42): connection-refused right after apiserver startup surfaces a raw OS error string instead of a friendly message. - bug-1676 (gitea #44): FROM_HOST always blank -- capnp schema has no from_host field, and OnJobUpdated's blind store overwrite wipes the apiserver-local value set at creation on the first controller push. Gitea issue #43 (EXEC_HOST lost after daemon restart) was NOT reproduced after two restart cycles with WAL-preserving state -- EXEC_HOST/state recovered correctly in every poll. Not logged as a bug; a related design gap (WAL_JOB_DISPATCH only carries {jobid, state}, not allocs) was flagged as an unconfirmed latent risk, not a live finding. bug-1677: scripts/gitea-sync.py's sync_bugs() reads buglog.json as a bare list but the real format is the {version, bugs: [...]} wrapper -- would crash on any real run. Found via static review while checking the gitea_issue cross-reference field convention; not executed live. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
parent
8fd232c198
commit
1c9cfc1f10
1 changed files with 104 additions and 0 deletions
|
|
@ -18444,6 +18444,110 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-23T21:03:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1673",
|
||||
"timestamp": "2026-08-24T13:40:00Z",
|
||||
"error_message": "Gitea issue #40 (\"job \u8c03\u5ea6\u4e0d\u6b63\u786e\"): `xcecli hosts list` shows MAX_SLOT=0 for a dynamically-joined worker even though the host's real NCPUS (e.g. 16 or 22) is correctly reported and scheduling correctly dispatches up to that many concurrent jobs. Reproduced live on branch scheduler-framework: single-node deploy, worker joined as 'worker-1', `xcecli hosts list` -> NCPUS=22 MAX_SLOT=0. Submitted 9 single-slot sleep jobs -- all 9 eventually dispatched and ran (no hard cap observed under this test), so the reporter's secondary claim of a hard ~8-job concurrency ceiling did NOT reproduce here (most likely a mid-dispatch snapshot, dispatch is not instantaneous across all pending jobs on this branch; a queue/usergroup slot limit in limits.yaml, e.g. ug1_normal_slots=8, remains a plausible alternate explanation worth asking the original reporter about). Only the MAX_SLOT=0 display value itself is a confirmed, reproducible bug.",
|
||||
"file": "src/frontend/apiserver/backend/handler.go, src/configs/hosts.yaml",
|
||||
"root_cause": "OnHostConfigUpdated() (handler.go:60-70) only applies config-pushed fields (including MaxSlot) onto a host entry via `store.Hosts.Get(incoming.HostName)` -- an EXACT hostname match. hosts.yaml commonly defines only a wildcard `hostname: default` entry (plus a couple of literally-named ones, e.g. worker2/host3) -- confirmed via grep, no entry named after the actual dynamically-joined worker (e.g. 'worker-1'). So a worker that joins under its own hostname never receives a HostConfig push under that literal name, `store.Hosts.Get()` misses, OnHostConfigUpdated returns early, and MaxSlot is left at Go's zero-value (0) forever -- completely decoupled from the value actually used for scheduling. The C-side path is correct: worker.c's apply_join_state()/find_matched_hostconfig() falls back to the 'default' wildcard entry's mxj/maxcpus for real dispatch decisions, confirmed via a live controller log line showing the correct slots=22 resource value. This is purely a Go apiserver display gap, not a scheduling defect.",
|
||||
"fix": "NOT FIXED -- reproduced and root-caused during a Gitea-issue verification pass (issues #40-44), no code change made. Candidate fix: OnHostConfigUpdated should fall back to matching the wildcard 'default' HostConfig entry (mirroring worker.c's own fallback) when no exact-named entry exists, instead of silently no-op'ing.",
|
||||
"tags": [
|
||||
"gitea-issue-40",
|
||||
"apiserver",
|
||||
"go",
|
||||
"handler.go",
|
||||
"hosts-list",
|
||||
"max-slot",
|
||||
"display-bug",
|
||||
"hostconfig"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1556"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-24T13:40:00Z",
|
||||
"gitea_issue": 40
|
||||
},
|
||||
{
|
||||
"id": "bug-1674",
|
||||
"timestamp": "2026-08-24T13:40:00Z",
|
||||
"error_message": "Gitea issue #41 (\"job output timestamp is incorrect\"): reporter's transcript shows `xcecli jobs list -l <jobid>` succeeding but omitting submit/start/done timestamps. Reproduced live on branch scheduler-framework, but the actual failure is more severe than reported: `xcecli jobs list -l <jobid>` (and bare `xcecli jobs list -l`) fail OUTRIGHT with `query request failed: [invalid parameters: unsupported query field: pending_reason]` -- no output is produced at all, not even a partial listing missing timestamps. Likely a regression introduced after the original issue was filed (this branch is under active scheduler-framework development).",
|
||||
"file": "src/frontend/apiserver/controllers/jobs.go:553, src/frontend/apiserver/controllers/jobs.go:901",
|
||||
"root_cause": "Both the `-l`/fields-filtered job-list code paths (jobs.go:553 and the near-duplicate at jobs.go:901) validate the requested field list via `utils.GetSupportedFields(models.JobSubmitRequest{})` -- the job SUBMISSION request struct -- instead of `models.JobInfo{}`, the job query/response struct (the correct pattern, confirmed already used correctly for hosts at hosts.go:105/307 with `models.HostInfo{}`, and for at least one other job code path at jobs.go:1131 with `models.JobInfo{}`). Read-only/response-only fields like `pending_reason`, `start_time`, `end_time`, `pids`, `pgids` don't exist on JobSubmitRequest, so the first such field hit in Go's randomized map-iteration order triggers a hard `unsupported query field` rejection, making `-l` (and any custom -o field list touching these fields) fail unconditionally.",
|
||||
"fix": "NOT FIXED -- reproduced and root-caused during a Gitea-issue verification pass, no code change made. Fix is mechanical: change both `utils.GetSupportedFields(models.JobSubmitRequest{})` call sites (jobs.go:553, jobs.go:901) to `utils.GetSupportedFields(models.JobInfo{})`, matching the working pattern already used at jobs.go:1131 and hosts.go's equivalent checks.",
|
||||
"tags": [
|
||||
"gitea-issue-41",
|
||||
"apiserver",
|
||||
"go",
|
||||
"jobs.go",
|
||||
"jobs-list",
|
||||
"field-validation",
|
||||
"regression"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-24T13:40:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1675",
|
||||
"timestamp": "2026-08-24T13:40:00Z",
|
||||
"error_message": "Gitea issue #42 (\"\u521a\u542f\u52a8\u7684\u65f6\u5019\u67e5\u8be2\u4f5c\u4e1a\u62a5\u7684\u9519\u8bef\u4e0d\u592a\u53cb\u597d\"): `xcecli jobs list` run in the narrow window right after the apiserver (re)starts, before its listener is up, prints a raw `get the request failed: [connection refused]` instead of a user-friendly message. Reproduced live on branch scheduler-framework: killed and relaunched the apiserver, hammered `xcecli jobs list` in a tight loop from the instant of relaunch -- attempts 1-13 all returned the exact reported text; the listener came up by attempt 14.",
|
||||
"file": "src/frontend/xcecli/common/request/request.go:97-98, src/frontend/xcecli/common/request/request.go:185-188",
|
||||
"root_cause": "ExtractErrorMessage() (request.go:185-188) explicitly special-cases the connection-refused case (`if strings.Contains(errStr, \"connection refused\") { return \"connection refused\" }`) but returns the raw OS-level string verbatim instead of a friendlier message (e.g. explaining the API server may still be starting up). GetRequestFromAPIServer() (request.go:97-98) then wraps it as `\"get the request failed: [%s]\"`, producing exactly the reported output.",
|
||||
"fix": "NOT FIXED -- reproduced and root-caused during a Gitea-issue verification pass, no code change made. Fix is localized: change the connection-refused branch in ExtractErrorMessage() to return an operator-friendly message (e.g. \"cannot reach the API server -- it may still be starting up, retry shortly\") instead of the bare OS error string.",
|
||||
"tags": [
|
||||
"gitea-issue-42",
|
||||
"xcecli",
|
||||
"go",
|
||||
"request.go",
|
||||
"error-message",
|
||||
"ux",
|
||||
"startup-window"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-24T13:40:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1676",
|
||||
"timestamp": "2026-08-24T13:40:00Z",
|
||||
"error_message": "Gitea issue #44 (\"FROM_HOST is -\"): `xcecli jobs list`'s FROM_HOST column always shows `-` instead of the host the job was submitted from. Reproduced live on branch scheduler-framework: submitted a job, checked `xcecli jobs list` ~1 second later (job still PEND) -- FROM_HOST blank; confirmed via the apiserver's own debug log showing `\"from_host\":\"\"` on every job regardless of state (PEND/RUN/DONE), across every job submitted in the session.",
|
||||
"file": "src/frontend/apiserver/controllers/jobs.go:1483, src/frontend/apiserver/backend/handler.go:18-20, src/schema/controller.capnp",
|
||||
"root_cause": "Three-part chain, fully traced: (1) jobs.go:1483 correctly sets `FromHost: ctx.ClientIP()` at job-CREATION time. (2) controller.capnp's Job/JobSubmitData structs have no from_host field at all (confirmed via grep -- zero matches for fromHost/from_host in the schema), so this value never round-trips to or back from the C controller. (3) handler.go's OnJobUpdated (line 18-20) does a blind `store.Jobs.Put(uint64(job.Unique_JobID), *job)` full overwrite on every controller-sourced job push, unlike OnHostUpdated/OnHostConfigUpdated (handler.go:46-70) which explicitly merge worker-sourced and config-sourced fields onto the existing stored entry. Since a controller-sourced job push never carries from_host (per #2), the very first state-change push after creation -- which happens almost immediately, well before a user can typically observe the initial value -- wipes FromHost back to \"\" via the blind overwrite.",
|
||||
"fix": "NOT FIXED -- reproduced and root-caused during a Gitea-issue verification pass, no code change made. A full fix needs either: (a) add a from_host field to controller.capnp's Job struct and thread it through the C controller so it round-trips and OnJobUpdated's overwrite naturally carries the correct value, or (b) the smaller fix of making OnJobUpdated merge-preserve FromHost from the existing stored entry the same way OnHostUpdated preserves config-sourced fields, at the cost of only ever reflecting the value set at creation time (never updated if resubmission semantics ever change).",
|
||||
"tags": [
|
||||
"gitea-issue-44",
|
||||
"apiserver",
|
||||
"go",
|
||||
"jobs.go",
|
||||
"handler.go",
|
||||
"from-host",
|
||||
"capnp-schema",
|
||||
"data-loss"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-24T13:40:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1677",
|
||||
"timestamp": "2026-08-24T13:40:00Z",
|
||||
"error_message": "scripts/gitea-sync.py's sync_bugs() would crash (or silently no-op) on any real invocation: `buglog = json.loads(BUGLOG_PATH.read_text())` (line 726) then `for i, bug in enumerate(buglog):` (line 731) treats buglog.json as a bare list, but its actual top-level structure is `{\"version\": 1, \"bugs\": [...]}` -- a dict. `enumerate()` over a dict yields its keys (\"version\", \"bugs\") as `bug`, and the very next line's `bug.get(\"id\", ...)` would raise AttributeError on the string \"version\". Discovered incidentally while checking whether to set the `gitea_issue` cross-reference field on newly-logged bugs (bug-1673..bug-1676) during a Gitea-issue verification pass -- not run live, found via static read of the sync script against the current buglog.json wrapper format.",
|
||||
"file": "scripts/gitea-sync.py",
|
||||
"root_cause": "sync_bugs() was apparently written against (or never updated from) an earlier bare-array buglog.json format. Every other buglog reader/writer in this session's own tooling (and the Stop hook's own validation, per .wolf/OPENWOLF.md's 'Always preserve the top-level {\"version\": 1, \"bugs\": [...]} wrapper') expects the wrapper; this script does not, unlike its own spec-sync half (sync_specs et al.) which was presumably tested more recently.",
|
||||
"fix": "NOT FIXED -- discovered via static review only, not executed live, no code change made. Fix is a one-line change: `buglog = json.loads(BUGLOG_PATH.read_text())[\"bugs\"]` (and the corresponding `_write_json(BUGLOG_PATH, buglog)` call at line 799 needs to re-wrap as `{\"version\": 1, \"bugs\": buglog}` rather than writing the bare list back out, or the wrapper will be destroyed on the next real sync run).",
|
||||
"tags": [
|
||||
"gitea-sync",
|
||||
"scripts",
|
||||
"python",
|
||||
"buglog-format",
|
||||
"latent-bug",
|
||||
"unrun"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-24T13:40:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue