RUNG ladder
Climbs the model catalog and the context length until the box is full. The answer is categorical before it is numerical: which model, at what context.
GB10 128 GB → Ling-3.0-flash at 131K
memory-aware benchmarks for agent fleets
How many agents can your box run before it clips?
Headroom is the margin before clipping. Agent Headroom replays real coding-agent sessions against your own hardware and measures five kinds of margin: size, speed, concurrency, models, and context.
headroom meter · 128 GB unified · planner estimate
12 agents fit · clips at 16
weights
KV cache
warm contexts
headroom
clipped
five tests · five margins
Each test answers one question with one number you can act on.
The workload is frozen: the same recorded agent sessions, replayed turn by turn with the same tool calls. Only the machine changes.
PACE inference
One agent, the full 168-turn replay at 65K context. Decode speed, end-to-end speed, and the wall clock a real session would take.
DGX Station → 88.8 tok/s, 8.1 min
CROWD host_pressure
Adds agents with live tools until the first turn fails. Host RAM and CPU are sampled the whole time, so you see what actually ran out.
Ryzen 192 GB → 2 agents, host pool full at 4
TENANTS residency
Keeps a set of models loaded at once and cycles work through them. Counts what stays resident and prices every swap in seconds.
GB10 → 3 resident, 79 s swap
RECALL warm_contexts
Parks each agent's context in host RAM between turns. How many agents stay warm before restores turn back into re-prefills?
128 GB RAM → 12 warm, collapse at 16
measured, not modeled
Same replay, same flags, five very different machines.
Every number below comes from a result file with the probe, the plan, the server command line, and a memory trace beside it.
Agents sustained with live tools
CROWD, Qwen3.8-27B at 65K. The first level where a turn drops ends the run.
The Ryzen box decodes as fast as the GB10 with one agent. Only the memory tests see the difference.
The fastest engine on the box
TRACKS, Nemotron 3.5 Lightning 30B-A3B on a GB10, 32K context, decode tok/s.
llama.cpp runs everywhere as the baseline row. The second row is what people really run on that platform.
Size headroom by machine
RUNG. Budget for weights plus KV in GiB, and the top rung each box reached.
The Station's bar is HBM plus LPDDR over NVLink-C2C: the BRIDGE badge. Scale is linear.
What a swap costs
TENANTS, four-model text preset. Residents stay loaded; Ling-3.0-flash is evicted and restored.
Restores are timed against how much of the file is still in page cache, so RAM size shows up here too.
one session drives the fleet
Probe, plan, run, report. Nothing to install on the targets beyond Python and a server.
The probe reads memory from the driver, never from comparing sizes. The planner only schedules what fits. The runner refuses anything the box changed since the plan.
bench probe
Read the machine
Topology, device and host memory, bandwidth, storage. Unified, discrete, coherent, or a cluster of nodes.
bench plan
Size the test
Picks the model, quant, context and placement from the catalog that fit this box, and says why anything was rejected.
bench run
Replay the agents
Launches the server with pinned flags, replays the recorded sessions, samples memory the whole time.
bench report
Fold the fleet
One table per test across every machine, including runs fetched back from remote hosts and clusters.
# on the box you are sitting at python3 -m bench probe python3 -m bench plan inference --model qwen3.8-27b --context 65536 python3 -m bench run out/plan-<host>-inference.json --replay agentperf-default-v1 # the same test on another machine in the fleet, results come back to out/remote/ python3 -m bench remote <host> probe python3 -m bench remote <host> run out/plan-<host>-host_pressure.json --levels 1,2,4,8,16 # fold everything into one report python3 -m bench report # markdown tables per test
badges
Three things most benchmarks skip.
BRIDGE
Coherent memory. On a DGX Station the GPU reads weights out of LPDDR over NVLink-C2C; the ladder measures how much of a 433 GiB model can live there and what it costs.
FLEET
Clusters. Several machines become one probe, one plan and one run, with the link between them measured, and a model split across nodes when none holds it alone.
TRACKS
Engine rows. llama.cpp on every machine for a fair comparison, plus vLLM, MLX or TensorFold where they are the fastest thing people actually run.
rules of the suite
Honest by construction.
Frozen workload
Recorded sessions from AgentPerf, replayed exactly. The same 2.5 million prompt tokens on every box, with the never-cacheable share fixed.
Memory from the driver
Topology comes from CUDA attributes, Metal, or sysfs, never from guessing by size. A 96 GB card next to 92 GB of RAM is still discrete.
Pinned artifacts
Every model file has a byte count and a SHA-256 in the catalog. A run on the wrong file says so in the result.
Failures are results
A dropped turn, an eviction that did not round-trip, a server that would not load: all recorded with the memory trace, not retried away.