Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):
1. WAL segment rotation was gated on a hardcoded 64MB compile-time
constant; trim_wal_segments() never deletes the active (currently-open)
segment. At soak job counts a segment never rotated, so nothing was
ever physically trimmable and ha_evaluate_repl_action always chose
STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
threading a wal_segment_bytes override through persist_config_t
(mirrors the --max-wal-records pattern: config.h default, persist.h
setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
--wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.
2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
(include/netio.h): the macro calls ep->msg_handler(ep) synchronously
(not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
endpoint_unref) could drop the endpoint's last ref and free it inside
the handler call, so the trailing write hit freed memory. This is the
first-ever live exercise of a node receiving a snapshot transfer while
mid-election. Fixed by holding a ref across the synchronous call, same
pattern task 5.8 established for the async path.
With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.
Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
Adds --max-wal-records (mirrors the existing --wal-sync-ms pattern:
main.c argp option -> myconf_t -> persist_config_set_max_wal_records())
so a deployment can force the WAL-record-count checkpoint trigger to
fire on a job count a soak can actually reach, instead of the
production default of 10000.
Adds a new `--force-snapshot` soak mode (run-ha-soak.sh): submits
enough jobs to cross the configured threshold, kills a non-primary
node, wipes its data dir (a genuine empty rejoin), restarts it, and
asserts the cluster reconverges with zero walcompare divergence and
zero job loss. Fails loud (not a false PASS) if the restarted node's
log shows no actual snapshot install, so the round can't silently
report success without exercising the real scenario.
INVESTIGATION FINDING (not yet resolved, this commit does not close
5.10a or ha-entity-snapshots 7.1): live-tested with temporary
diagnostic logging (since reverted). schedule_checkpoint() correctly
fires once wal_record_count reaches the configured threshold
(confirmed: "wal_record_count=100 max=100" logged), but the delaymgr-
scheduled checkpoint_task_fn never executes afterward -- no checkpoint
ever completes, no snapshot file is ever written, in a live multi-node
deployment. This is a separate, deeper issue in the delaymgr scheduling
path (or its wiring for this specific call shape), not a bug in this
commit's CLI plumbing or soak scaffolding -- both are built, tested,
and confirmed working up to the point where the async checkpoint
should run. Needs its own dedicated investigation (likely src/lib
delaymgr, not persist.c) before --force-snapshot can produce a PASS.
This also has production implications beyond the test scenario: if
confirmed, WAL checkpoints may never fire automatically today, meaning
WAL segments are never trimmed via this path in a live deployment.
controller_unit_test: test_ha_startup 34/0, test_ha_core_safety 28/0,
test_persist_checkpoint 5/0, test_walcompare 17/0 (no regression).
Soak --selftest 17/17.
The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.
Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.
- endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
at refcnt=1 (the owner ref).
- endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
- vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
read ep->busy on any queued vtask without the ep being freed underneath it.
- every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
raw endpoint_fini+free. Several of those also leaked the ep struct before
(endpoint_fini-only); unref fixes the leak too.
Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)
Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
Root-caused via the non-ASAN debug build + gdb: on a chaotic restart an inbound
endpoint dispatches a request (e.g. walCatchupEntry op 37) while ep->owner
(cluster_t) is NULL — not yet wired at accept, or cleared during teardown. The
handler did `c = ep->owner` then `ha_dispatch_gate(&c->ha, ...)`; with c==NULL,
&c->ha = NULL+offsetof(cluster,ha) = 0x3ff0, a garbage pointer ha_dispatch_gate
dereferenced (atomic_load &ha->role) -> SIGSEGV, dropping the node below quorum
and stalling the election.
Guard c==NULL after decode (so the decoded request is freed, no leak): log a
loud WARN and drop the request — there is no cluster context to dispatch through
and no c->myhost to address a reply. The controller stays alive (never
self-terminate on an operational condition); the peer retries once the endpoint
is owned. Backtrace: ha_dispatch_gate(ha=0x3ff0) <- myreq_handler cluster.c <-
vtask_entry <- run_task.
Second of the two round-4 blockers (bug-1282 deadlock already fixed). Re-running
the 7.2 chaotic soak with both fixes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Lands the PeerCtrlDispatch machinery behind a DEFAULT-OFF flag
(g_drive_async_peerctrl_dispatch_enabled); production behavior is
UNCHANGED. peer_ctrl_msg_handler becomes a thin shim that takes
ownership of ep->fsm_in.pkt and either runs the routing body inline
(flag OFF, byte-for-byte the prior path) or dispatches it to the vtask
pool via vtask_enq_classified (flag ON). The 3 routers now take pkt as
an explicit param (no cross-thread ep->fsm_in.pkt re-read). Per-endpoint
arrival order is enforced by bind_vtask's endpoint_block_tasks.
Packet ownership: shim clears ep->fsm_in.pkt at dispatch (sole-owner
transfer); peer_ctrl_vtask_entry NULLs iod->pkt before the body frees
it once. New vtask_t.owns_io_pkt (default 0) lets taskq_item_killer free
the carried packet for an un-executed queued task at vtaskmgr_fini WITHOUT
double-freeing the libxce endpoint-owned path (bug-1199).
Adversarial review found two FLAG-ON hazards, documented with loud
comments (flag is OFF so production is unaffected), to be resolved before
task 3.4 flips the flag ON:
- bug-1197 (CRITICAL): raft_cb_become_follower endpoint_fini()s peer
vote-eps from a pool worker, racing the io_worker's read of ep->data;
needs iomgr_unmonitor before endpoint_fini in every pool-reachable
teardown.
- bug-1198 (HIGH): peer_ctrl_classify tags all work VTASK_LIVENESS, but
the CTRL_PRIMARY repl-ack path can run a blocking snapshot checkpoint
inline; resolved by task 3.2 (offload snapshot to SlowLane).
Per spec, 3.1 is not 'done' until the task 3.4 deadlock gate is green.
Tests: 3 new peerctrl tests; controller 196/196, lib 282/0; 0 net leaks.
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
Dedicated-pthread pool isolated from the vtask pool; non-blocking slowlane_submit
(0 enqueued / -EEXIST deduped / -EAGAIN cap-full-deferred), admission cap on
running tasks, per-key in-flight dedup, run-exactly-once, drain+join fini. Wired
into controller lifecycle (dormant; consumers land in 2.x/3.x). 7 unit tests,
mutation-verified cap+dedup; full suite 194/0 green, 15x ASAN.
drive_async_liveness_reserved = max(1, min(n_peers-1, n_vcpus-1)) — single
source of truth for VTASK_LIVENESS_RESERVED (deadlock-freedom floor applied
last). New DEFAULT_CONNECT_TIMEOUT_MS / DEFAULT_SLOWLANE_* constants. Resolved
reservation logged at init; under-provisioned (n_vcpus<n_peers) clamp WARNs
loudly, never self-terminates. 8 unit cases, mutation-verified.
1.1: WalCatchupEntry gains term@4/prevLogIndex@5/prevLogTerm@6 (append-only).
1.2: RequestVote/PreVote gain lastLogTerm@4/lastLogIndex@5; cluster.c vote callers
fixed to the extended library arity (RESTORES xce_controller + controller_unit_test
compile+link, broken by the raft-commit-safety signature extension); ha.c emits the
candidate's last-log tuple on send.
5 minimal stub log/commit callbacks wired into s_real_cbs so raft_init's new
non-NULL requirement is satisfied — tasks 2.x replace them with real bodies; the
logOk freshness gate is inert until the term seam (1.3) + on_get_entry_term (2.2).
Pre-existing test_repl_unit.c wal_catchup -EAGAIN failures + ASAN leaks are an
overlayfs /tmp fsync env artifact (bug-850), not from this task (untouched paths;
append-only schema cannot affect the @3 entries-list decode). Build/baseline
isolation is blocked by a c-capnproto flake-hash drift — see Implementation Notes.
Append a committed-consume branch to the OP_nop handler (existing liveness/reply
byte-unchanged): ha_consume_heartbeat_committed applies the non-primary gate
(3.4) + stale-term guard (accept hb->term >= ha->term, 3.2), then delegates to
the monotonic ha_observe_heartbeat_committed (1.2). Absent body / zero / failed
gate are no-ops (4.2). No vote-floor edit — safety is the term-guard. 4 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Desync FAILOVER re-elections by node priority (ha_election_stagger_ms /
ha_failover_stagger_fn) mirroring the startup stagger, with seed jitter and
async prevote start; allow prevote in voter roles through the dispatch gate.
Adds HA role and vote-gate tests (stagger desync by priority, prevote on dead
primary, stickiness-lapse grants).
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
The cluster.c denial-reason elimination ordered catch_up before fresh_heartbeat,
disagreeing with raft_cb_vote_eligible (which checks fresh-heartbeat-on-STANDBY
before catch-up). In the rare STANDBY catch-up + fresh-heartbeat overlap the log
could report catch_up while the gate denied on fresh_heartbeat. Expose the pure
read predicate ha_heartbeat_is_fresh and mirror the gate's exact order so the
logged reason can no longer disagree with the actual denial. Decision unchanged;
log-string fidelity only. prevote/vote-gate/failover-prevote suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add OPCode_preVoteRequest case in cluster.c, mirroring the real-vote case:
freshness floor + reused eligibility/stickiness gate (raft_on_pre_vote_request),
reply over preVoteReply with the voter's UNCHANGED term (no adoption), denial
logging with candidate id + reason. Pure insertion; real-vote case untouched.
6 new vote-gate tests assert deny-while-leader-alive, grant-when-lapsed, and the
no-mutation/term-unchanged contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cluster_start (which set c->running) is orphaned on the HA myapp_run path, so
the running flag was never set on HA nodes — check_controller() treated the
primary as shutting down and zeroed joblist/workerlist reply chunks. Own the
flag lifecycle in cluster_run, symmetric with the c->running=0 on shutdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live symptom: jobsub intermittently failed with 'Connection timed out' even
though the cluster had quorum. Root cause: the synchronous commit barrier waits
only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was
not configurable. 50ms is too tight for durable sync majority replication —
especially since standbys now fdatasync each replicated entry before ACK — so a
multi-entry jobsub's later WAL entry intermittently misses the window:
commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50)
-> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT.
Fix:
- Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync
majority replication, still bounded for failure detection.
- Add a --sync-timeout-ms controller flag threaded arguments -> myconf ->
persist pcfg (override applied before persist_init); 0 = use default, positive
overrides — so operators can tune per environment.
TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All
timeout-sensitive tests already pass an explicit timeout_ms (none rely on the
default), so none are slowed; full controller suite green under ASAN. No
commit-engine/barrier/replication/wire change beyond the default value + plumbing.
bug-561.
Root cause (found via ha-convergence-recovery task 8 live 3-node E2E): a
controller with no local config (config_version_tag==0) could become PRIMARY
and push an empty config, breaking replication.
Two upstream election-safety gaps, both outside ha-convergence-recovery:
1. RDHUP re-election bypassed the no-config 'voter only' guard. The startup
path (ha_start_election_async) refuses to start an election when
my_config_tag==0 && n_peers>1, but the RDHUP failover path
(ha_on_primary_ctrl_fd_rdhup) called raft_on_leader_lost unconditionally.
Fix: mirror the guard before elect_ctx alloc / hook / raft_on_leader_lost;
a config-less node stays CTRL_STANDBY (voter only) and grants the
configured peer instead of starting (and winning) its own election.
2. Voter freshness gate compared the candidate only against the voter's LOCAL
config, ignoring the cluster's known committed state. Fix: ha_vote_self_freshness
floors the comparison at max(local, learned-committed config_tag / committed_lsn),
so a voter denies any candidate whose config/cluster state is outdated vs
what the cluster has committed. Strictly additive (never denies the freshest
node — Leader Completeness preserves liveness).
TDD: test_rdhup_noconfig_node_does_not_elect + configured-node contrast;
test_ha_vote_gate learned-floor deny + liveness + backward-compat. Full
controller suite green under ASAN. commit.c and ha-convergence-recovery
files untouched.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
cluster_t embeds commit_engine_t; init in cluster_init_phase2 after the
vtaskmgr guard (idempotence-guarded — phase2 can run twice), wiring
repl_set_commit_engine + ha.repl back-pointer before any standby can
register; fini after iomgr/vtask shutdown. persist_wal_sync_append
passes written_lsn; engine-wired path drops -ENODEV and the
committed_lsn store (engine owns advancement); LSN-0 first-entry
routes to legacy per the accepted interim. p2 -ENODEV pin updated to
the Req 6.3 contract. 34/34 + 6/6 green through honest delegation;
census exactly at the 3 validated baselines; production binary links.
Standby sends WalAckReply via xce_reply after each successfully
applied WAL entry. ha_apply_wal_catchup_entry signature extended
with endpoint_t* and cluster_t* parameters. walackreplymsg union
arm added to RequestReply capnp schema and xce_reply_t. Error
entries skip ACK. Dispatch gate test confirms OPCode_walCatchupEntry
allowed in CTRL_STANDBY/ELECTING/INIT.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
OPCode_nop handler now sends stateReport {applied_lsn, config_tag}
instead of nodata. Function in ha.c for unit-test linkability.
Two tests: warm state and zero-state fresh standby (Req 1.4).
Replace stub with ha_apply_wal_catchup_entry() calling apply_wal_entry().
CAS-advance applied_lsn on success; log ERROR and skip on failure.
Standby stays in CTRL_STANDBY throughout. 3 new unit tests.
Enforce heartbeat_timeout_ms >= 2 * heartbeat_interval_ms at startup.
validate_heartbeat_config() in startup_validation.c returns -EINVAL with
error message on violation; wired in cluster_init_phase1 before ha_init().
4 unit tests added to test_ha_startup.c.
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
Extract cluster_init_managers() which initializes configmgr, jobmgr,
resmgr, workermgr, qmgr, sharedinst, xstrings, endpoints, and job_obj.
Call it from cluster_init_phase1 (after init_controller_tunables) so
all managers exist before election.
This fixes the heap-buffer-overflow in handle_connection: incoming
OP_requestVote SSL connections arrive during the pre-election phase
and call zarray_store(&c->endpoints) — which requires endpoints to
be initialized first.
cluster_init_phase2 is simplified: it no longer re-initializes these
managers and only sets up iomgr/SSL, opens the listener, and loads
cluster-config-dependent data.
Add ctrl_tunables struct to cluster_t with per-table fields:
jobmgr_nhash/delta, workermgr_nhash/delta, qmgr_nhash/delta,
resmgr_nhash/delta/inst, xstrings_nhash
init_controller_tunables() sets defaults (jobs=512, others=128)
and is called from cluster_init_phase1 so values are available
before cluster_init_phase2. Phase2 reads from c->tunables instead
of cluster_config or local #defines.
Each table gets its own values since they hold different item counts:
jobs can scale to hundreds, workers/queues/resources are typically
single-digit to tens.
Replace cluster_config-dependent hash sizes with compile-time constants
(CTRL_PERF_NHASH=512, CTRL_PERF_DELTA=64, CTRL_PERF_INST=64) for:
jobmgr, workermgr, qmgr, resmgr, xstrings
These are controller performance tuning parameters (capnp fields @13–@17)
that have no config generation tool yet. Using hardcoded defaults unblocks
cluster startup without requiring these fields in the config blob.
TODO: wire back to cluster_config once config generation tool is ready.
1. iomgr_monitor double-register: cluster_start skips iomgr_monitor
if c->ep was already registered by cluster_register_listener
(ep->flags non-zero after successful registration). Previously
returned EEXIST → -1 → "Operation not permitted" crash.
2. ASAN stack-use-after-return: cluster_fini now calls iomgr_stop()
before iomgr_fini() to join the worker thread before the cluster_t
stack frame is reclaimed by the caller.
3. Election timing: add ha_wait_for_heartbeat_or_timeout() before
election in myapp_run. If an existing primary sends a heartbeat,
node becomes STANDBY immediately without competing in election.
If timeout expires (no primary running), proceed to election.
This prevents ctrl-a from immediately losing when ctrl-b/c are
not yet started — it now waits election_timeout_ms (2s) first.
vote_request_thread connects to peers via SSL. iomgr was started before
election but c->ep was only registered with epoll inside cluster_start(),
which runs after election. The TCP connect succeeded (kernel accepts it)
but SSL_accept was never called server-side, so SSL_connect failed
immediately — both vote threads went to deny in <1ms and ctrl-a could
never win a quorum.
Fix: add cluster_register_listener() that wires c->ep into iomgr's
epoll watch. Call it from myapp_run() right after iomgr_start/vtaskmgr_start,
before election. cluster_start() still calls iomgr_monitor again
(idempotent via EPOLL_CTL_MOD).
The vote_request_thread was sending raw capnp(RequestVoteRequest) bytes
but the server-side iomgr decodes every incoming packet as a Request
struct. This caused the term field (uint64) to be misread as the opcode
field (uint16), so vote RPCs were silently misrouted (term=1 → OPCode_reply,
term=2 → OPCode_hello) and no node ever got a vote from its peers.
Fix:
- controller.capnp: add requestvotemsg @27 to Request.body and
requestvotereplymsg @14 to RequestReply.data
- typemap.h: move request_vote_req_t/reply_t before xce_request_t;
add requestvotemsg/requestvotereplymsg to body/data unions
- ha.c: rewrite vote_request_thread send/recv to use xce_encode()
wrapper and decode outer Request→inner RequestReply on receive
- cluster.c: add case OPCode_requestVote handler calling
handle_request_vote() and replying via xce_reply()
All 188 unit tests pass.
