tryConnect built the xce_client_config_t with C.CString(host/from/key), then a
defer freed all three. But xcecli_ctx_init stores the config POINTER
(ctx->conf = config), not a copy — client_failover.c dereferences
config->masters/from/key for the client's ENTIRE lifetime (e.g.
xce_failover_next_target strncpy's a master host on every write-fail/redirect).
Freeing the strings left config->masters[0] dangling → ASAN heap-use-after-free
(strncpy in the background C thread) after a few submits.
Fix: keep the three C strings for the process lifetime (tryConnect runs once at
startup, gCtx is a process-global); free them only on the init-failure path.
Verified: 6 consecutive job submits, apiserver stays up, 0 ASAN errors
(previously crashed after ~2-3 submits).
Co-Authored-By: Claude <noreply@anthropic.com>
modelJobToSubmit built the C job_startinfo_t without setting execuser, so the
controller forwarded a NULL execuser in the startjob request and the worker
ran every job as "nobody". Now the submit carries the resolved exec user: an
explicit exec_user wins, otherwise the job runs as the submitting user.
- add resolveExecUser() (plain-Go, unit-testable) + TestResolveExecUser
- set si.execuser in modelJobToSubmit, free it in freeJobSubmit
Note: deploying this exposed an ASAN heap-use-after-free in the apiserver's
submit/notification path (strncpy reading freed memory); the fix is committed
but needs the memory-lifecycle investigation before it can ship safely.
Co-Authored-By: Claude <noreply@anthropic.com>
controller.capnp declared `struct Data $C.mapname("xce_data_t")` (an
OPCode_get query-result tagged union) at file scope, which shadows
Cap'n Proto's builtin Data blob type for every later `: Data` field in
the same file. This silently miscompiled 7 fields across 6 structs as
a struct-pointer type instead of a raw byte blob: UpgradeConfigRequest
.sha256/.bytes, ReplConfigReply.sha256/.bytes, ReplConfigPushRequest
.sha256/.bytes, ReplEnvelope.payload, SnapshotTransfer's chunk
payload, WalCatchupEntry.payload/.batchPayloads, and CtrlHelloRequest
.key. Production C code had already routed around it for years via
hand-rolled raw capn_ptr manipulation (see request.c's own comments);
nothing before now needed the generated codec/accessors to actually
work for these fields.
Renamed the local struct to XceData (capnp-level name only; the
$C.mapname("xce_data_t") C typedef is unchanged) and explicitly
repointed getdata/pushmsg at XceData by name, since those two fields
correctly want the local tagged union. Updated every derived
generated-identifier reference across 19 hand-written files spanning
controller, lib, scheduler, and the Go apiserver/xcecli CGo bindings.
Fixed 5 call sites in lib/request.c that had been typed against the
old workaround alias but now legitimately receive the correct
capn_data type (a pure type-name swap -- Data_ptr/XceData_ptr and
capn_data share the identical {capn_ptr p} layout).
Verified via full rebuild+test: lib (1/1), controller (3/3, incl. the
job-array push regression tests), scheduler (15/15), Go apiserver +
xcecli (all packages green), and a direct pycapnp round-trip proving
UpgradeConfigRequest.sha256/.bytes now encode/decode correctly.
Bug: bug-1461 (.wolf/buglog.json). Unblocks admin-config-tool task 3.3.
Cross-daemon integration test, test-only, spanning the scheduler (C)
and API-server (Go/CGO) receive paths built in tasks 5.1-5.4/5.3b.
Drives real xce_encode()/xce_decode() wire bytes through the actual
production dispatch chain on both daemons -- extract_data() ->
scheduler_req_handler() -> sched_handle_dataset() ->
sched_on_job_array() and go_push_handler() -> handleDataset() ->
cJobArrayToModels()/dispatchJobArray() -- skipping only vtask_entry()'s
~10 lines of static glue, mirrored verbatim in test code (the
if (callnext) treats-any-nonzero-as-proceed behavior that caused
5.3b's bug is reproduced exactly).
Redelivery is tested via genuinely independent fresh decodes of
separately-encoded bytes, not a decode-once-call-twice shortcut.
Bug-catching capability proven by temporarily reverting task 5.3b's
fix and confirming both the C and Go tests fail meaningfully, then
restoring to green -- confirmed independently by both the implementer
and reviewer.
Task 5.5 of job-array-single-wal-record (Requirements 6.1-6.5, 8.4).
This completes Phase 5 (Compact Scheduler/API-Server Push).
Adds the Go/CGO receive path for Data.jobArray, mirroring task 5.3's
scheduler-side C implementation: rawJobArrayToModels() (pure-Go
base_jobid + i expansion), cJobArrayToModels() (CGO adapter sibling to
cJobToModel), dispatchJobArray() (dedup via the same store.Jobs.Has()
the ordinary Data_data_jobs path already writes through, single
confirmation log only when at least one element was newly
dispatched), and a new case C.Data_data_jobArray in handleDataset.
Elements route through OnJobUpdated, matching the ordinary single-job
push's own routing. The CGO boundary needed no changes -- both
C.xce_job_submit_t and C.Data_data_jobArray were already exposed from
prior tasks, including task 5.3b's extract_data() fix which this
receive path depends on identically to the scheduler's.
store.ArrayJobs (the array-level rollup aggregate) is deliberately
left unpopulated -- confirmed out of this spec's scope, deferred to a
future spec per requirements.md's Boundary Context.
Task 5.4 of job-array-single-wal-record (Requirements 6.2, 6.4, 6.5, 8.4).
- parsePeer: use net.SplitHostPort (handles IPv6, stdlib convention)
- tryConnect: take Go strings, own CGO allocation internally
- RegisterToBackend: remove unreachable 'no HA candidates' error path
- Sync Peers field comment between utils and backend config structs
- COR-001: drain runHeartbeat goroutine via done channel before test returns,
eliminating race against concurrent SetHeartbeatInterval tests
- COR-002: replace time.After with time.NewTimer + defer t.Stop() + t.Reset()
to prevent timer goroutine leaks on context cancellation
- COR-005: guard HeartbeatInterval store write with > 0 check in OnClusterUpdated,
keeping store consistent with SetHeartbeatInterval's zero-ignore semantics
- add heartbeat.go with heartbeatIntervalSecs var, SetHeartbeatInterval, runHeartbeat (no build tag, CGO-free)
- add heartbeat_test.go with 5 tests covering interval validation and goroutine lifecycle
- add StartHeartbeat to clib.go (CGO impl calling xcecli_heartbeat)
- remove SetHeartbeatInterval stub from stub.go (now real code in heartbeat.go)
- remove TestSetHeartbeatInterval_StubNoOp from stub_funcs_test.go
- call SetHeartbeatInterval in OnClusterUpdated to track backend-pushed interval
- call backend.StartHeartbeat in cmd/main.go after backend registration
Add OPCode_notify handler in go_push_handler top-level switch
(not inside handlePush) to receive resource value updates from
the controller via xce_notify_data_t/schedUpdateResvals.
- Add 8 CGO union accessors for xce_val_t (which, i32, i64,
u32, u64, fval, str, bool) since CGO cannot access union fields
- Add handleNotifyData() dispatching NotifyData_data_schedUpdateResvals
events, iterating xce_resval_list_t and calling cResvalToDistribution
- Add cResvalToDistribution in convert.go with 7 separate switch
cases for Value_u_* union discriminants (no combining i32/i64)
- Add rawResvalData + rawResvalToDistribution pure-Go version
in convert_pure.go for testability without CGO
- Add 10 TDD tests covering all 7 value types, nil val guard,
unknown discriminant, and boolean false
- All 42 backend tests passing
Add four function stubs to backend/stub.go for the !cgo build path:
- StartHeartbeat(ctx context.Context) — no-op (heartbeat not available)
- SetHeartbeatInterval(secs int) — no-op (interval update not available)
- EvalSelectForHost(hostname, selectExpr string) error — returns errNoCGO
- Cleanup() — no-op (no C resources to release without CGO)
TDD: four tests in stub_funcs_test.go verify stubs behave correctly
(no-op functions don't panic, error stubs return errNoCGO). All backend
tests passing.
1. Add //go:build !cgo to control_host_test.go so the stub-testing
tests only run in non-CGO builds, matching the stub.go build
constraint. Clean up dead nil check in SignatureCompiles test.
2. Replace two independent if-statements for actionType mapping in
ControlHost with a map lookup + explicit guard, consistent with
ControlQueue's ActionMap pattern. Prevents silent zero-value
actionType if SupportedHostActions expands later.
- backend/clib.go: Replace ControlHost stub with real implementation
using xcecli_hostctrl, following the same pattern as ControlQueue
- controllers/hosts.go: Remove package-level allFlag variable;
preCheckHostControlRequest now returns (bool, error) so the allFlag
is a local value in the caller, eliminating the data race
- controllers/queues.go: Same fix for preCheckQueueControlRequest
- Add unit tests for both preCheck functions verifying return values,
error cases, and concurrent safety
- Add compile-time verification test for ControlHost stub signature
Start from existing entry and apply this-source's fields, rather than
starting from incoming and restoring the other-source's fields. The new
pattern makes it immediately clear which fields each push type WRITES,
without having to reason about which fields are being "restored".
OnHostUpdated keeps the merged := *incoming + if ok guard because the
worker push also populates HostModel/HostType from attrs; config-sourced
values must only win when a prior config push has already set them.
All 37 backend unit tests pass.
Wires the remaining cluster_config_t data into the notification pipeline:
- dispatchConfigPush() extracted as a pure-Go helper (CGO-free, testable)
that drives OnQueueUpdated, OnResourceUpdated, OnHostConfigUpdated, and
OnClusterUpdated from a single config push event
- handleConfigPush() delegates to dispatchConfigPush via the new helper,
eliminating the previous inline loops
- OnHostConfigUpdated and OnClusterUpdated added to NotificationHandler
interface and implemented with partial-merge semantics in DefaultHandler
- rawHostConfigData/rawHostConfigToModel and rawClusterData/rawClusterToModel
added to convert_pure.go with their CGO extraction counterparts in convert.go
Safety fixes from code review:
- OnHostUpdated, OnHostConfigUpdated, OnClusterUpdated now copy *incoming to
a local merged value before mutation, closing the TOCTOU window between
store.Get and store.Put under concurrent pushes
- cClusterConfQueues and cClusterConfResources gain nil-pointer guards
matching the existing guard in cClusterConfHostConfigs
- handleConfigPush gains an explicit gHandler nil check
- cClusterConfToClusterInfo returns nil on empty clustername (prevents phantom
empty-key write to store.Clusters); handleConfigPush logs a warning when nil
- 36 unit tests passing (CGO_ENABLED=0)
