Commit graph

30 commits

Author SHA1 Message Date
lei.chen
85a22271c1 Add X11 forwarding for interactive jobs (xcecli -I -X, ssh -X style)
Render -X jobs' windows on the user's own X server with zero client
install, mirroring the ssh -X experience: the job command starts with
DISPLAY pointed at a per-job TCP listener on the worker, and the user's
xcecli relays that listener's byte stream to their local X server.

- xcecli: new -X/--x11 flag (interactive=2 + exdata "x11":1). After
  submit it fetches the per-job MIT-MAGIC-COOKIE-1, registers it in the
  user's Xauthority, then bridges one xattach websocket to the local X
  server address parsed from $DISPLAY (unix socket or TCP). The cookie
  entry is removed when the session ends. Pre-flight checks report
  missing DISPLAY/xauth before submit.
- apiserver: GET /xce/v1/jobs/:id/xattach relays websocket binary frames
  to the worker's forwarding port (raw TCP, no ttyd). Close codes match
  the ttyd attach (4000 unreachable, 4001 ended). The job-finished event
  ends the relay. Teardown closes both hops as soon as either pump
  exits, so a client disconnect propagates instead of deadlocking in a
  blocking read.
- worker: XJOB jobs with exdata x11=1 skip VNC, listen on 6100-6199,
  generate a per-job xauth file (/tmp/job_<id>_xauth, worker-side since
  the worker has no session), export DISPLAY/XAUTHORITY to the command,
  and bridge the app connection (loopback) with the relay connection
  (non-loopback; roles classified by source address, not arrival order).
  A 60 s no-client watchdog reaps jobs nothing ever attached to; the
  v1 single-connection boundary refuses a second X connection loudly.

t1-verified per the plan matrix: window rendering (xclock/xeyes on
Xvfb), normal end -> DONE + xauth removed, no-client 60 s reap (137),
kill -> immediate teardown with no residue, -I/-I --browse regression,
daemon health, missing-DISPLAY error, second-connection refusal.
(--xjob VNC on t1 fails for lack of a vncserver binary; pre-existing.)

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-17 02:16:06 +00:00
lei.chen
3545aa0d77 apiserver: apply the controller's jobPidInfo start-report push
handleNotifyData had no case for NotifyData_data_jobPidInfo, so the
controller's start-report push (apiserver_job_start_fun, fired from
HOOK_DETAIL_JOB_START the moment WAL_JOB_START commits) was silently
dropped. interactive_job_url then stayed invisible until the next
periodic rusage-driven job UPDATE re-delivered the full row — up to
10s later (job_rusage_collect_interval), the dominant chunk of an
-I --tty job's submit-to-first-output latency (measured ~6s on t1,
server-side work itself is ~0.6s).

Add OnJobPidInfo to the notification interface and DefaultHandler:
it merges the worker-reported pids/pgids and the exdata {"url": ...}
onto the already-stored job record (get-mutate-put, mirroring
OnJobFinished) and publishes a job.updated event. The CGO decode
side is factored into cPidInfoToRaw/pidInfoJSONB, shared with the
existing xce_job_t conversion.

Cluster-verified on t1: URL is pollable ~0.6s after submit
(previously ~11s), full submit/run/finish lifecycle clean, all
daemons healthy.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-15 10:08:23 +00:00
lei.chen
74121eb72e fix(apiserver): map pidinfo from job start into the job model (interactive_job_url)
The worker packs the interactive session URL (ttyd/XJOB) into the start
report's exdata as {"url": ...}; the controller stores it on
job->job.pidinfo->exdata and the capnp Job wire carries pidinfo (pidinfo@9,
JobPIDInfo.exdata@3) — but apiserver's cJobToModel never mapped pidinfo, so
models.JobInfo.InteractiveJobURL (declared only in the query-field whitelist
JobAdditionalFields) had no producer and interactive_job_url was permanently
empty in API responses. xcecli's waitForInteractiveURL/openBrowser flow could
never fire.

- models.JobInfo: add InteractiveJobURL (json interactive_job_url, matching
  the JobAdditionalFields whitelist name).
- convert_pure.go: rawPidInfo (pids/pgids/exdata) + interactiveURLFromExdata;
  rawJobToModel marshals pids/pgids into PIDInfo (pid_info is a JSONBJFields
  entry, parsed by processJobData) and promotes the exdata URL to
  InteractiveJobURL.
- convert.go: cJobToModel extracts job.pidinfo via pidinfo_pid/pidinfo_pgid
  cgo helpers (CGO cannot index the C arrays directly).

Cluster-verified end to end on t1 (fresh apiserver deployed): tty job
(xcecli -I --tty equivalent, exdata {"interactive":1}) now returns
interactive_job_url=http://<worker>:9000 and a populated pid_info; non-tty
jobs return ""/no pid_info. All daemons healthy, zero errors in window.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-14 09:20:27 +00:00
lei.chen
0c8607345e merge: sync Rongsong's scheduler-framework (c5b172c7..3442a57d, 781 commits)
Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).

Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.

Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
  queue refuses new submissions only; pending jobs still run) --
  supersedes our QUEUE_CLOSED preason on the dispatch path;
  sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
  rounds (sched_stash_config/sched_apply_pending_config), applied at
  the same main-thread round boundary that preserves bug-1513
  serialization
- bug-1760: sched_check_pending_rejoin exposed +
  xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
  (superset of our bug-1500 execuser fallback; our job-typed
  resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
  secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
  branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
  src/harness/test.c; controller suite split into
  controller_test.c + fixtures; their clitest fixtures spawn xce_test
  mid-suite under `meson test -j`

Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
  gate re-ported into their relocated allocator (sched_election.c)
  after plugin_job_election, with attach on the allocator thread
  only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
  merge initially re-carried our first-detail-add-promotes-main_reason
  fallback in add_job_preason (4e2ac86), but their whole trace/verify
  stack treats "detail-only => main_reason 0" as a contract --
  sched_trace_capture counts main_reason as its own occurrence
  ("Expected 2 Was 3" in test_trace_capture), the outcome renderer
  prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
  and the committed .replay/.sim_verify baselines encode main_reason=0
  (test_repl_trace_baseline byte compare, test_trace_equivalence
  off-vs-traced). The fallback is dropped from the shared lib;
  the user-visible "overall reason" survives as an API-layer view:
  rawJobToModel (convert_pure.go) promotes the first detail row to
  main_reason_name when main_reason is 0 (main_reason VALUE marshals
  unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
  (their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
  resend, xcecli_force_reconnect + their is_connection_broken both
  exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
  /xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)

bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
  (testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
  harness_test.c -- hundreds of cases per process) call
  case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
  dirs are swept immediately in tearDown (after engine reaping), FAILED
  cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
  disables. One full controller_unit_test run was observed to leak 12G+
  to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
  failures
- harness_unit_test_deps links the shim library as well

Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
  INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
  covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
  bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
  RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
  terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
  by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
  detail-keeps-main-0 contract; new tests cover limit-code detail
  rows and explicit-main-wins-over-later-details; apiserver
  convert_test.go gains the detail-only name-promotion cases

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-14 05:17:03 +00:00
367af11af1
fix(worker): graceful job kill and catchable stop, as LSF (bug-1957)
A plain kill sent one SIGTERM and never escalated, and a plain stop sent
one uncatchable SIGSTOP. Following LSF's TERMINATE and SUSPEND actions, a
plain kill now sends SIGINT, then SIGTERM, then SIGKILL, and a plain stop
sends SIGTSTP (the job can catch it, prepare and stop itself), then SIGSTOP
if it has not stopped; each next signal follows after the grace period and
only while the job is alive (or not yet stopped). A resume cancels a
pending stop; the job's exit ends any sequence. An explicit signal is
delivered once; a force kill is SIGKILL at once.

- JobSignal gains grace (capnp @2, xce_signal_t.grace): > 0 asks for the
  sequence, in seconds between signals.
- apiserver: job.terminate_interval (default 10 s, LSF's
  JOB_TERMINATE_INTERVAL); jobCtrlParams sends kill = SIGKILL and stop =
  SIGSTOP with that grace when no signal is given; the handler no longer
  pre-fills a default signal.
- worker: grace travels through the job-control task, the kept-signal table
  (pending_signal_*_ex) and the IPC message; the job process runs the
  sequence (job_signal_seq_*, uv timer) and checks stop state in /proc.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 10:00:36 +00:00
148557a5d5
fix(worker): deliver the job-control signal the user asked for (bug-1954)
worker_signal_callback() sent SIGTERM for a KILL request without force and
SIGSTOP for any TSTP request. job_signal_to_deliver() (new job_signal.c) now
delivers the requested signal; only a forced terminate escalates to SIGKILL.
The apiserver's default signal for a plain stop becomes SIGSTOP (it was
TSTP and relied on the worker's substitution), so a plain stop still
suspends jobs that ignore SIGTSTP.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 09:07:38 +00:00
4d2e80c4bb
fix(apiserver): send the requested signal, force kill and checkpoint options (bug-1940)
Cause: ControlJob (backend/clib.go) looked its content argument up as a
signal name, but the job control handlers pass a JSON JobControlEvent, so
the lookup always missed and every kill sent TERM (stop TSTP, resume
CONT); sig.force and ckpt.kill were hard-coded to 0 and the checkpoint
method was never set.

Fix: new pure jobCtrlParams (backend/convert_pure.go) decodes the event
(or a bare signal name/number), validates the signal against
JobSignalMapTable and returns an error instead of falling back, and maps
force_kill to xce_signal_t.force (kill) or xce_ckpt_options_t.kill
(checkpoint) plus the checkpoint method. ControlJob fills the C structs
from it. Tests: backend/jobctrl_params_test.go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 08:43:53 +00:00
d01ced9bd0
fix(apiserver): send exdata, exec cwd and exec home with submitted jobs (bug-1936)
Cause: modelJobToSubmit() filled only cmdline, execuser, launch_type and
pre/post-exec of job_startinfo_t, so the extended data the submit handler
builds (env, *_limit incl. -W run limit, file transfers, interactive mode),
the cwd and the exec home never reached the controller or the worker.

Fix: map the start info in a pure submitStartInfo() (exdata from
ExtendData, execcwd from ExecCWD or CWD, exechome from ExecHome) that
modelJobToSubmit copies into the C struct; decode exdata back in
cJobToModel/cJobArrayToModels so the store keeps ExtendData and CWD.
Reject extend_data the worker's parse_exdata() cannot take (not a JSON
object, over EXDATA_LENGTH-1 bytes) and drop an empty transfer list
instead of sending a null.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 08:38:26 +00:00
1cedd7f791
fix(controller,apiserver,xcecli): expose job CPU peak and efficiency to the apiserver and xcecli (bug-1736)
The controller computed run usage (CPU time, memory, CPU peak and
efficiency, LSF bjobs -l definitions) but the capnp Job struct had no
field for it, so the job pushed to the apiserver never carried it, and
snapshots (the same struct) dropped it: a job restored from a checkpoint
lost its accumulated usage.

- controller.capnp: Job.runUsage @13 (JobRunusage); old entries decode
  with no run usage.
- apiserver: map run_usage into JobInfo; max_mem, avg_mem, used_mem,
  cpu_time, peak_cpu, peak_cpu_time, peak_cpu_duration, mem_efficiency,
  avg_cpu_efficiency, peak_cpu_efficiency become queryable columns.
- xcecli: -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK,
  CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now
  renders too), and a MEMORY USAGE / CPU USAGE block in jobs list -l.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 13:54:43 +00:00
b88e352616
feat(job-array-one-based-index): apiserver backend ID helpers and array ID wrapper (task 5.1)
GenerateArrayJobID wraps libxce idgen_array (first element, index 1;
one-element arrays included) with a non-cgo stub; pure helpers
ParseHumanJobIDStrict (written identity with exact field-width checks,
uint64 overflow reported as a width error), IsArrayCapableKey and
MakeElementID (bit-identical to libxce jobsel_element_id).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 11:29:11 +00:00
099dbdabdc
fix(apiserver): fix job control union discriminant, add human-readable job ID support (gitea#45)
ControlJob() set xce_jobctrl_t.action to the job_ctrl_type_t action code
(kill/stop/resume/checkpoint), but that field is actually the capnp
JobControl_u_which union discriminant -- a different enum entirely. This
made kill/resume/checkpoint hit request.c's -EINVAL default case, and made
stop collide with the checkpoint discriminant and read through the wrong
union member. Live-verified against a real deployed cluster: kill/stop/
resume/checkpoint now behave correctly with no apiserver crash.

Also additively accept a human-readable job ID ("{cluster}-{apiserver}-
{timestamp}-{seq}") on the job control endpoint, alongside the existing
numeric form xcecli already translates to -- lets a caller hit the REST
API directly without reimplementing xcecli's ID packing. Live-verified via
a direct curl call with the human-readable ID.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
2026-09-07 08:31:54 +00:00
5ab344e1ae
fix(apiserver): propagate submit user as job execuser (bug-1780)
modelJobToSubmit() never set job_startinfo_t.execuser when converting
a submitted job to the C struct sent to the controller/worker, so
every job ran as the worker's 'nobody' default and failed to exec
(exit 126). Default execuser to the submitting user via a new
resolveExecUser() pure helper, and free the newly allocated C string.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
2026-09-07 08:10:10 +00:00
e395eaa3ad
fix(frontend): count host job-state columns and EXEC_HOST by real slots
NJOBS/RUN/SSUSP/USUSP (xcecli hosts list -l) and EXEC_HOST (xcecli jobs
list) always showed a hardcoded 1 slot per host regardless of a job's
actual per-host allocation -- rawJobToModel()'s ExecHosts JSONB builder
never read the real per-host "slots" resource from the C job allocation,
so the already-correct downstream slot-counting logic (computeHostSlotUsage,
bug-1743) and EXEC_HOST's "N*host" formatter never had real data to work
with. convert.go now extracts each host's real slots allocation via a new
CGO indexing helper, mirroring the scheduler's own get_nslots_in_alloc()
convention.

Also wires hosts list -l's "CURRENT LOAD USED FOR SCHEDULING" table (tmp/
swap/mem/slots and friends) to the host's real live resource values --
previously entirely hardcoded placeholder strings, never connected to any
backend data.
2026-09-07 07:32:00 +00:00
lei.chen
4e894d2184 merge: sync Rongsong's scheduler-framework into chatbot
Full merge of origin/scheduler-framework (scheduler-core restructure,
failover-adoption, WAL_JOB_DISPATCH, apiserver backend health/heartbeat,
build-time versioning, bug-1673/1676/1678/1679/1680).

Conflict resolution:
- scheduler: adopt Rongsong's scheduler-core architecture (driver loop,
  queue project/picker fields, resource bug-1569 fix)
- worker/controller_connector.c: keep Rongsong's reconnect/rejoin
  structure, PORT our bug-1509 resend_job_status onto the rejoin success
  path (his reconnect didn't re-report job state)
- apiserver backend: take Rongsong's health/heartbeat (supersedes our
  e505066 circuit breaker)
- group.c: keep our bug-1513 comment
- misc: take theirs for additive fixes (go_build versioning,
  test_group.c, persist.c RUSAGE_UPDATE, FilterRows bug-1680)

NOTE: our scheduler fixes (bug-1494 family dispatch-fail requeue,
PEND-reason classification) are NOT in Rongsong's branch — needs
verification/porting onto scheduler-core (separate follow-up).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-25 12:33:16 +00:00
1a61b04ab6
fix(apiserver): thread from_host through the CGO conversion layer (bug-1676, gitea #44)
modelJobToSubmit/freeJobSubmit (submit direction) and
cJobToModel/cJobArrayToModels (receive direction, both single-job and
array-job paths) now carry from_host across the CGO boundary.
rawSubmitData/rawJobToModel (the pure-Go, CGO-free mapping layer) gain
the field too, completing the round-trip so JobInfo.FromHost reflects
whatever the controller actually has -- which, as of the prior two
commits, is now always the client-asserted hostname. backend/handler.go
needs no change: its existing blind overwrite in OnJobUpdated is
correct by construction once every controller push genuinely carries
the right value.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
2026-08-24 15:14:49 +00:00
lei.chen
4e2ac861f4 feat(scheduler): PEND reason classification — main_reason + limit/queue/worker reasons (POC "why is my job waiting")
P0 (done, cluster-verified):
- main_reason was always 0: add_job_preason only set it when BOTH host and res
  were NULL, but the allocator always attaches host/res detail rows. Now the
  first detail add wins as main_reason, so the API's main_reason_name is
  meaningful (was "unknown(0)"). Verified live: PEND jobs now report
  main_reason=3/match_fail + per-host reason_name.
- limit plugin attach: host-based quota hit (hg1_slots_per_host) -> LIMIT_HOST
  + limit name; global quota hit (ug1_slots, ...) -> LIMIT_GLOBAL + limit name.
  Previously both blocks were silent (job just stayed PEND).
- Extended the PREASON_* enum (was 3 codes) with LSF-style classes:
  LIMIT_GLOBAL/LIMIT_HOST/QUEUE_CLOSED/QUEUE_DRAINING/TIME_WINDOW/HOST_UNAVAIL.
  apiserver pendReasonName maps them; host/res detail rows now carry reason_name.

P1 (implemented):
- queue status: sched_queue_is_ready no longer a stub — closed/inactivated
  queues are not electable; the fcfs election attaches QUEUE_CLOSED /
  QUEUE_DRAINING to every PEND job in a skipped queue.
- worker filter: sched_try_alloc_on_host_ skips UNAVAIL/CLOSED workers and
  attaches HOST_UNAVAIL instead of trying to allocate on them.

Unit tests: preason suite updated for the main_reason change + 3 new tests
(limit host/global, first-add-wins); 295 lib tests pass under ASAN.
apiserver backend tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-14 09:43:21 +00:00
lei.chen
ac6131dbd7 feat(apiserver): expose PEND reason end-to-end (scheduler → API)
The pending_reason field was declared on models.JobInfo but never populated:
scheduler computed PEND reasons and the controller stored them on
job->pend_reasons, yet the data stopped there. Two gaps closed:

1. controller/apiserver.c: apiserver_job_hook_fun now handles
   HOOK_DETAIL_SETPREASON by pushing the updated job to API servers (reusing
   apiserver_job_update_fun). Without this, a job whose PEND reasons change
   never reached the API server — only JOB_NEW/UPDATE/START/EXIT events did,
   and setpreason fired none of those.

2. backend/convert.go + convert_pure.go: cJobToModel / rawJobToModel now
   extract xce_job_t.pend_reasons (main_reason + host/res reasons) and
   marshal them into models.JobInfo.PendingReason as JSONB. models/jobs.go:
   pending_reason added to JOB_DEFAULT_COLUMN so default job queries return
   it (it was only in JSONBJFields, never selected).

Verified live on t1/t2/t3: a job with resreq "type == nosuchhost" stays
PEND and returns
  "pending_reason":{"main_reason":0,"host_reasons":[{"host":"t1","reason":3},
   {"host":"t2","reason":3},{"host":"t3","reason":3}]}
(main_reason=3 PREASON_MATCH_FAIL per host) via both /jobs list, /jobs/:id
and the SSE stream. 4 new unit tests (rawJobToModel pendReason marshal +
name mapping), all apiserver backend tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-13 09:16:13 +00:00
lei.chen
c90e63ec0f fix(frontend): job control no longer returns HTTP 500 (bug-1490)
Two stacked defects in the apiserver job-control link, both of which kept
hold/release/cancel from ever reaching the controller:

1. controllers/jobs.go: validateJobControl and foundJob asserted
   jobinfo.Data["array_idx"].(int64), but StoreToOrderMaps produces
   array_idx as int (models.JobInfo.ArrayIdx is declared int) -> panic
   -> Gin Recovery -> HTTP 500 before backend.ControlJob ran. Add
   normalizeJobInt64 / jobArrayIdx helpers and use them in both
   functions (job_status already had this fix via normalizeJobStatus;
   array_idx was missed).

2. backend/clib.go: backend.ControlJob set xce_jobctrl_t.action to the
   models.JOB_* constant (JOB_STOP=1 / JOB_RESUME=2 / JOB_KILL=3 /
   JOB_CHKPNT=8), but that field is the capnp JobControl union
   discriminant (JobControl_u_siginfo=0 / JobControl_u_ckptoption=1).
   The controller's job_ctrl() selected the wrong union branch (or
   none, since encode_JobControl drops the union pointer for
   non-0/1 discriminants) and rejected every control with -EINVAL.
   Add jobCtrlDiscriminant() and use it.

Also: resolveExecUser was defined in the cgo-only convert.go but is
exercised by a non-cgo test, so `make test` (CGO_ENABLED=0) failed to
build the backend package. Move it to convert_pure.go so it compiles in
both modes.

Tests: 3 controller regression tests (validateJobControl allowed /
not-allowed branches, foundJob array-child branch) + 1 backend
jobCtrlDiscriminant test. make test fully green. Cluster E2E verified:
hold -> PSUSP, release -> PEND, cancel -> EXIT on a PEND job, no 500,
no -EINVAL, all daemons healthy.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 07:35:51 +00:00
ff1be48001 merge 2026-07-23 12:35:10 +08:00
Lei Chen
3a2094a461 Merge branch 'working' into job_peek_lchen
job_peek_lchen: Add fetchfile feature (job output/error peek proxy)
working: HA/Raft infrastructure (consensus, replication, config mgmt)

Conflict resolution summary:
- controller.capnp: fetchfile @41, fetchfilereply @18, fetchfilemsg @30
- cluster.c: bifurcated OPCode_reply for fetchfile + config-push ACK
- job.c: kept full fetchfile pending-request subsystem (~183 lines)
- request.c: kept both xce_fetchfile_req() and XCE_RETRY_DEFAULTS
- helper.c: added missing preVoteRequest/preVoteReply mappings
- All 15 new test registrations (test_cbuf.c + 14 HA tests)
2026-07-22 13:57:25 +08:00
9e0d87fe50 feat(job-array-single-wal-record): API-server-side (Go) local expansion of the compact push
Adds the Go/CGO receive path for Data.jobArray, mirroring task 5.3's
scheduler-side C implementation: rawJobArrayToModels() (pure-Go
base_jobid + i expansion), cJobArrayToModels() (CGO adapter sibling to
cJobToModel), dispatchJobArray() (dedup via the same store.Jobs.Has()
the ordinary Data_data_jobs path already writes through, single
confirmation log only when at least one element was newly
dispatched), and a new case C.Data_data_jobArray in handleDataset.

Elements route through OnJobUpdated, matching the ordinary single-job
push's own routing. The CGO boundary needed no changes -- both
C.xce_job_submit_t and C.Data_data_jobArray were already exposed from
prior tasks, including task 5.3b's extract_data() fix which this
receive path depends on identically to the scheduler's.

store.ArrayJobs (the array-level rollup aggregate) is deliberately
left unpopulated -- confirmed out of this spec's scope, deferred to a
future spec per requirements.md's Boundary Context.

Task 5.4 of job-array-single-wal-record (Requirements 6.2, 6.4, 6.5, 8.4).
2026-07-15 15:56:48 +00:00
d47b3375e7 controller will pass resource <name, value> pair than <idx, value> pair 2026-05-13 07:18:03 +00:00
Lei Chen
d2b47e3985 Init commit of jobs output/error message peek project. 2026-05-09 16:57:54 +08:00
ce2edf0dac refactor: represent time fields as int64 Unix seconds throughout the stack
Collapses a four-step conversion chain (time_t → int64 → time.Time →
RFC3339 string → string → time.Time) to a direct int64 pass-through:

  C time_t → rawJobData int64 → JobInfo int64 → JSON integer → QueryJob int64 → time.Unix() for display

Changes:
- apiserver/models/jobs.go: SubmitTime, QueueTime, StartTime, EndTime,
  ReserveTime in JobInfo and JobAdditionalFields: time.Time → int64
- apiserver/models/common.go: ActionControl.SubmitTime: time.Time → int64
- backend/convert_pure.go: remove time.Unix() calls; assign int64 directly
- backend/convert_test.go: update time assertions to compare int64 values
- xcecli/models/jobs.go: SubmitTime, StartTime, EndTime, ReserveTime in
  QueryJob: string/time.Time → int64; remove time import
- xcecli/common/output/time.go: replace parseJobTime(string) with
  unixToTime(int64) and fmtUnixTime(int64) helpers
- xcecli/common/output/page_print_jobs.go: replace parseJobTime() calls
  with unixToTime()/fmtUnixTime(); remove !startTime.IsZero() guard
  (unixToTime returns false for ts<=0, making it redundant)
- xcecli/common/output/page_print_array_elements_jobs.go: replace
  string-split+time.Parse submit-time block with fmtUnixTime(); remove
  unused time import

Side effect: eliminates the "0001-01-01T00:00:00Z" zero-time
serialization issue at the source — zero timestamps are now 0 (integer)
and unixToTime/fmtUnixTime return false/"-" for ts<=0 without special-
casing needed in callers.
2026-05-09 07:00:49 +00:00
3ed7ef011a fix go jobId schema bug 2026-05-03 11:45:14 +08:00
d7afd72e5b fix front end issues 2026-05-01 21:22:45 +08:00
bc3103a6d5 feat: P1-1 handle resource value push via OPCode_notify
Add OPCode_notify handler in go_push_handler top-level switch
(not inside handlePush) to receive resource value updates from
the controller via xce_notify_data_t/schedUpdateResvals.

- Add 8 CGO union accessors for xce_val_t (which, i32, i64,
  u32, u64, fval, str, bool) since CGO cannot access union fields
- Add handleNotifyData() dispatching NotifyData_data_schedUpdateResvals
  events, iterating xce_resval_list_t and calling cResvalToDistribution
- Add cResvalToDistribution in convert.go with 7 separate switch
  cases for Value_u_* union discriminants (no combining i32/i64)
- Add rawResvalData + rawResvalToDistribution pure-Go version
  in convert_pure.go for testability without CGO
- Add 10 TDD tests covering all 7 value types, nil val guard,
  unknown discriminant, and boolean false
- All 42 backend tests passing
2026-04-17 13:54:03 +00:00
893812e7aa feat: P0-1e expand handleConfigPush with host config and cluster dispatch
Wires the remaining cluster_config_t data into the notification pipeline:
- dispatchConfigPush() extracted as a pure-Go helper (CGO-free, testable)
  that drives OnQueueUpdated, OnResourceUpdated, OnHostConfigUpdated, and
  OnClusterUpdated from a single config push event
- handleConfigPush() delegates to dispatchConfigPush via the new helper,
  eliminating the previous inline loops
- OnHostConfigUpdated and OnClusterUpdated added to NotificationHandler
  interface and implemented with partial-merge semantics in DefaultHandler
- rawHostConfigData/rawHostConfigToModel and rawClusterData/rawClusterToModel
  added to convert_pure.go with their CGO extraction counterparts in convert.go

Safety fixes from code review:
- OnHostUpdated, OnHostConfigUpdated, OnClusterUpdated now copy *incoming to
  a local merged value before mutation, closing the TOCTOU window between
  store.Get and store.Put under concurrent pushes
- cClusterConfQueues and cClusterConfResources gain nil-pointer guards
  matching the existing guard in cClusterConfHostConfigs
- handleConfigPush gains an explicit gHandler nil check
- cClusterConfToClusterInfo returns nil on empty clustername (prevents phantom
  empty-key write to store.Clusters); handleConfigPush logs a warning when nil
- 36 unit tests passing (CGO_ENABLED=0)
2026-04-16 11:25:03 +00:00
960ef62342 fix job array display issue 2026-04-14 18:23:49 +08:00
c9dc8c4676 Merge Go frontend (apiserver + xcecli) into meson build 2026-04-14 11:23:18 +08:00