myxce/src/docs/apiserver-backend-gaps.md

API Server Backend Gap Analysis

Source: src/controller/testsuite/test.c (C test tool) vs src/frontend/apiserver/ (Go API server)

Overview

The C test tool (test.c) directly calls xcecli_* C library functions to interact with the controller. The Go API server currently operates in a push-based, read-only-from-store model: the controller pushes data (jobs, workers, queues, resources) via Cap'n Proto, and the API server reads from its in-memory stores. Write/control operations are forwarded via the CGO backend layer, but only a subset of the C API is wired up.

---

Currently Implemented (Backend Forwards to C Controller)

C Test Command	C API Call	Go Backend Function	REST Endpoint
jobsub	xcecli_jobsubmit()	backend.SubmitJob()	POST /xce/v1/jobs/create
jobkill/jobckpt	xcecli_jobctrl()	backend.ControlJob()	POST /xce/v1/jobs/:jobids/control
qctrl	xcecli_queuectrl()	backend.ControlQueue()	POST /xce/v1/queues/control
(partial)	xcecli_hostctrl() (N/A)	backend.ControlHost() stub only	POST /xce/v1/hosts/control

Push notifications (read path, already working via handlePush):

• Job updates via OnJobUpdated -> store.Jobs
• Worker/host updates via OnHostUpdated -> store.Hosts
• Queue updates via OnQueueUpdated -> store.Queues
• Resource definitions via OnResourceUpdated -> store.Resources
• Resource locations via OnResourceLocationUpdated -> store.ResourceLocations

---

NOT Implemented -- Backend Request Gaps

1. Resource Value Query (cmd_resval -> xcecli_resval())

• What it does: Queries current numeric/string values of a named resource on a specific host (or cluster-wide). Returns typed values (float, string, bool).
• REST impact: No GET /xce/v1/resources/values endpoint exists. The resource controller only serves definitions and distributions, not live values.
• Priority: High -- Essential for monitoring dashboards to show real-time mem, r1m, slots etc.

[feedback] api server will not pull resources from controller, it just waits controller to push updated resource information, so api server just handle resource information update will be OK.

2. Resource Value Update (cmd_resupdate -> xcecli_update_resvals())

• What it does: Worker pushes updated resource values to the controller.
• REST impact: No equivalent. Currently workers call the C API directly. If the API server should support an admin endpoint to update resource values (e.g., for testing or override), this is missing.
• Priority: Low-Medium -- Workers typically call this directly; rarely needed via REST. Could be useful for admin overrides.

[feedback] This item will not be applied to api server, resource update function need to be used to update resources information of worker node(such as cpu/memory/..., and api server will not talke workload, so it do not need this)

3. Job Dispatch (cmd_jobrun -> xcecli_jobdispatch())

• What it does: Dispatches a job to specific hosts with allocation info (host:ntasks pairs). This is how the scheduler tells the controller to start running a job on specific workers.
• REST impact: Not exposed. The POST /xce/v1/jobs/create handles submission, but dispatch (assignment to hosts) is a separate operation currently done by the scheduler directly.
• Priority: Low -- Dispatch is an internal scheduler->controller operation. Not typically user-facing.

[feedback] treat this as low priority, IBM LSF has a brun command which issued by user to force a job to run in given host, we may need this. also cmd_jobrun() has been used to test execution flow/protocol between controller and work daemon

4. Job Info / Single Job Detail (cmd_jobinfo -> xcecli_jobinfo())

• What it does: Queries detailed info for a single job by ID, including pend reasons.
• REST impact: The GET /xce/v1/jobs/:jobids endpoint reads from the in-memory store, which is populated by push notifications. This works for already-known jobs but cannot query a job that hasn't been pushed yet. The C xcecli_jobinfo() is a synchronous request-response to the controller -- fundamentally different from the store-read pattern.
• Priority: Medium -- Store-based reads are sufficient for most cases. The sync query path would only matter if pushes are delayed or the store is stale.

[feedback] no need this, as job information will be pushed to api server

5. Cluster Configuration (cmd_getconf -> xcecli_clusterconf())

• What it does: Synchronously fetches the full cluster config (resources, queues, workers, host groups, user groups, global configs).
• REST impact: Currently the API server relies entirely on push notifications to populate stores. There's no way to do a one-shot full sync with the controller. On startup, it joins via xcecli_join(HostRole_apiserver) but doesn't explicitly request the current state.
• Priority: High -- Without an initial sync, if the API server restarts mid-session, it loses all state until the next push. The store is stale until a new push arrives.

[feedback] api server need handle pushed cluster configuration

6. Host Control (backend.ControlHost() -> xcecli_hostctrl())

• What it does: The C library doesn't expose xcecli_hostctrl() yet (the Go stub returns an error).
• REST impact: POST /xce/v1/hosts/control exists but will always fail: "host control not yet supported via C library".
• Priority: High -- Users cannot open/close hosts via the API.

[feedback] we need this

7. Set Pending Reason (cmd_setpend -> xcecli_set_pend_reason())

• What it does: Scheduler sets pend reasons on a job (host-based or resource-based).
• REST impact: No equivalent endpoint. Job pend reasons are currently populated only via push data.
• Priority: Low -- This is a scheduler->controller internal operation.

[feedback] set pending reason is for scheduler to set why job will be pend, this is used for test scheduler to controller job pending api

8. Run Tasks (cmd_runtasks -> xcecli_runtask())

• What it does: Launches individual tasks on specific hosts (not a full job, but sub-task execution).
• REST impact: No endpoint exists.
• Priority: Low-Medium -- Used for advanced task launch scenarios. Could be needed for MPI job management.

[feedback] this function is used by worker to run non rank 0 tasks(this is for MPI jobs) and api server need not handle this

9. Shutdown (cmd_shutdown -> xcecli_shutdown())

• What it does: Sends shutdown command to the cluster controller.
• REST impact: No endpoint. Currently cluster management operations (shutdown, restart) are not available via REST.
• Priority: Medium -- Operations teams need remote cluster control capability.

[feedback] we may need to admin tool, which talk to controller directly for this, as normal user do not need this at all

10. Resource Requirement Select (FilterHostByResReqSelect)

• What it does: The hosts POST /select endpoint exists but returns empty results -- it's a stub: processHostData([]utils.OrderMap{}, "").
• REST impact: POST /xce/v1/hosts/select is defined but non-functional.
• Priority: Medium -- This is the XCE equivalent of lsload -R / host matching by resource requirements. Important for job submission workflows to find eligible hosts.

[feedback] we may need this

---

Enhancement Proposal (Prioritized)

Priority	Enhancement	Effort	Description
P0	Initial state sync via xcecli_clusterconf()	Medium	On startup (after RegisterToBackend), call xcecli_clusterconf() to do a full state sync into stores. Prevents stale stores after API server restart.
P0	Implement ControlHost() in C library + Go wrapper	Medium	Expose xcecli_hostctrl() in client.h, then replace the Go stub in clib.go with a real CGO call.
P1	Resource value query endpoint	Medium	Add GET /xce/v1/resources/values?host=X&name=Y -> backend.QueryResourceValues() -> xcecli_resval(). This is a synchronous request-response pattern (different from push).
P1	Implement FilterHostByResReqSelect	Medium-High	Wire POST /xce/v1/hosts/select to evaluate resource requirement expressions against current host state from the store.
P2	Cluster shutdown endpoint	Low	Add POST /xce/v1/clusters/control -> backend.ShutdownCluster() -> xcecli_shutdown(). Should be admin-only with RBAC.
P2	Resource value update endpoint	Low	Add POST /xce/v1/resources/values/update -> backend.UpdateResourceValues() -> xcecli_update_resvals(). Admin/testing use.
P3	Task run endpoint	Low-Medium	Add POST /xce/v1/tasks/run -> backend.RunTask() -> xcecli_runtask(). For advanced MPI task management.
P3	Set pending reason endpoint	Low	Add POST /xce/v1/jobs/:jobid/pending-reason -> backend.SetPendReason() -> xcecli_set_pend_reason(). Internal/scheduler-facing only.

---

Architecture Note: Synchronous vs Push-Based

The biggest design decision lurking here: several of these C APIs (xcecli_resval(), xcecli_clusterconf(), xcecli_jobinfo()) are synchronous request-response calls -- the client sends a request and blocks until the controller replies. This is fundamentally different from the current push-based pattern where the controller asynchronously pushes updates.

To implement these, we'd need to either:

1. Direct CGO call (simpler but blocks the Go goroutine -- acceptable if calls are fast and we set timeouts)
2. Request-response via push channel (more complex -- send request, register a wait, receive push reply)

Recommendation: option 1 for initial implementation (it's what SubmitJob and ControlJob already do), with timeouts and goroutine isolation to prevent a slow controller from blocking the API server.

---

Additional C API Calls Not Used Anywhere (Reference)

These exist in client.h but are not called by test.c either:

C API Call	Purpose
xcecli_hello()	Health check / greeting
xcecli_nop()	No-op heartbeat
xcecli_heartbeat()	Periodic heartbeat
xcecli_notify_jobstart()	Worker notifies job started (with PID info)
xcecli_notify_jobfail()	Worker notifies job failed
xcecli_notify_jobstop()	Worker notifies job stopped
xcecli_notify_jobresume()	Worker notifies job resumed
xcecli_notify_jobpostdone()	Worker notifies job post-exec done
xcecli_update_jusage()	Worker updates job resource usage
xcecli_notify()	Generic notification
xcecli_notify_taskstart()	Worker notifies task started
xcecli_notify_taskfail()	Worker notifies task failed
xcecli_notify_taskfinish()	Worker notifies task finished
xcecli_clear_reason()	Clear pend reasons for jobs
xcecli_block_tasks()	Block task processing
xcecli_unblock_tasks()	Unblock task processing
xcecli_get_my_endpoint()	Get local endpoint info
xcecli_get_myinfo()	Get local peer info

[feedback] I think we just need heartbeat to tell controller that I'm alive, and other functions for worker and scheduler

These are primarily worker->controller notifications. They would be needed if the API server were to act as a proxy for worker daemons, which is not the current design.