TL;DR for operators
A serverless image platform must handle many bursty workflows that reuse some models and adapters but differ in computation, loading cost, and latency requirements. The usual assumption is that each customized workflow should scale as one indivisible GPU deployment.
ServerlessT2I shows why that unit can be too coarse. In its production trace, median GPU compute utilization reached 100%, while memory use was only 16 GiB at the median and 35 GiB at the 95th percentile. Compute was saturated, but substantial memory remained available. Monolithic deployment strands that capacity because the platform cannot independently scale, place, or cache the reusable stages inside each workflow.
By treating an image request as a chain of reusable model stages, the platform can manage those stages separately and use spare GPU memory to reduce loading and tensor-transfer costs. In trace-driven NVIDIA H800 tests, the design sustained up to twice the request rate, required as few as one-third the GPUs at a fixed rate, and handled tighter deadlines and greater burstiness than the evaluated monolithic variants.
For operators, the decision is no longer only how many complete workflows to replicate. It is which components need capacity, how GPU service should be divided across heterogeneous tenants, and whether provisioning or billing should reflect the actual work performed. The reported gains apply to the tested platform, hardware, workloads, and baselines rather than serving as universal guarantees.
A saturated GPU can still contain stranded capacity
The production trace presents an unusual resource picture. Median GPU streaming-multiprocessor utilization reaches 100%, yet median memory use is only 16 GiB and the 95th percentile is 35 GiB. Compute is busy while much of GPU memory remains unused. Meanwhile, 75% of T2I functions have an inter-arrival-time coefficient of variation above one, indicating bursty demand.
A monolithic serverless deployment cannot exploit this imbalance well. It packages text encoders, diffusion models, decoders, ControlNet, LoRA adapters, and their dependencies as one opaque function. If one stage becomes the bottleneck, the platform replicates the whole workflow. Shared models remain hidden behind separate deployment units, and parallel execution requires customer-managed placement and tensor movement.
The relevant capacity unit is therefore smaller than the workflow container: one image request is a chain of model invocations whose stages need not share the same scaling, placement, or caching policy.
ModelFns expose the workflow the platform needs to manage
ServerlessT2I introduces a ModelFn for each model stage. It declares inputs, outputs, loading, and execution; the orchestrator then derives a directed acyclic graph from those dependencies.
Once the graph is visible, the platform can scale a reused encoder without duplicating every downstream model, place independent branches in parallel, and account for service at stage level. Tensor movement becomes a runtime responsibility rather than customer-written infrastructure.
This is not a new image-generation model, and it does not claim better image quality. The evaluation keeps inference computation unchanged and reports identical images for matched prompts and seeds. Nor is it merely a whole-workflow cache. Its contribution is exposing model and layer boundaries that monolithic serving conceals.
Slack memory pays the cost of decomposition
Splitting a workflow creates two costs: more model loading and more inter-GPU tensor movement. ServerlessT2I uses spare GPU memory to absorb both.
For weights, it profiles each model and caches the smallest prefix of early layers needed to hide later host-to-GPU loading behind computation. In a Flux1-Schnell microbenchmark, this reduced loading latency by 90% and end-to-end latency by 21% relative to layer-wise pipelined loading without pre-cached layers. This is mechanism evidence, not the full end-to-end result.
For communication, NVSHMEM-backed transfers and callback-triggered fetching overlap tensor movement with producer and consumer work. Cached weights are evicted at layer granularity.
Inference buffers, communication regions, and cached weights still compete for physical memory. Model weight virtualization maps logical layers across PyTorch and NVSHMEM allocations instead of fixing rigid partitions. The associated sizing tests are sensitivity evidence: oversized communication regions increased loaded-weight volume by up to threefold and reduced SLO attainment by as much as 50%, while undersized regions risked stalls. With virtualization, the reported runs observed no allocation stalls, retries, or out-of-memory events at 97% peak memory utilization.
Fairness needs a GPU-service meter
Equal request counts are not fair when workflows differ in computation, loading, and transfer costs. ServerlessT2I measures cumulative wall-clock GPU service as vTime, including all three.
Scheduling has two stages. First, tenants remain eligible while their accumulated service is within an operator-set fairness slack. Second, ready ModelFns are ranked using waiting time, estimated execution latency, and remaining critical-path work. The formal service-difference bound applies only to tenants that remain continuously eligible and backlogged.
The scheduler experiment tests a policy tradeoff. SLO attainment rose from 62% under strict fairness to 84% at slack 20, while service divergence remained governed by the configured bound. The operator chooses a position on the fairness-efficiency curve rather than eliminating it.
What the benchmark evidence establishes
The main evaluation replays scaled production demand across five basic workflows in Setting S1 and twenty basic or adapter-augmented workflows in Setting S2. The default physical testbed contains 32 NVIDIA H800 GPUs; baselines are serverless adaptations of vLLM-Omni and Diffusers under shared workloads and LRU caching.
| Evidence | Likely purpose | Operational reading | Boundary |
|---|---|---|---|
| Up to 2× sustainable request rate at 90% SLO attainment | Main end-to-end evidence | More request capacity from the same GPU budget | Maximum across tested settings |
| 8 GPUs versus 24 for the strongest vLLM-Omni baseline in S1 | Main resource-efficiency evidence | Threefold GPU-count difference at one operating point | Basic-workflow setting |
| 90% lower loading latency from prefix caching | Component microbenchmark | Small resident prefixes can remove most loading from the critical path | Flux1-Schnell test |
| About 3% average DAG execution overhead | Implementation-cost test | Coordination cost is modest relative to reported gains | Setting S1 |
| 3.3% execution-time overhead at 256 GPUs | Scale extension | Overheads remain limited in the calibrated model | Simulation, not physical deployment |
Together, the tests support a coherent mechanism: graph-level management improves reuse and placement; the data plane controls decomposition costs; measured GPU service governs heterogeneous tenants. They do not show that one feature alone produces every end-to-end gain.
Business interpretation: capacity, billing, and onboarding
What the paper directly shows: within the tested H800 and NVSHMEM environment, model-granular serving improves SLO attainment, GPU utilization, burst tolerance, and request capacity relative to the evaluated monolithic variants.
Cognaptus inference: cloud operators could convert those gains into lower infrastructure cost, higher accepted load, or less burst-driven over-provisioning. Creative platforms with many LoRA and ControlNet combinations may benefit because shared stages need not become separate capacity islands. The declarative interface may also reduce customer engineering for placement, parallelism, and tensor transport.
vTime has a commercial implication beyond scheduling. Request-based quotas or prices can misrepresent heterogeneous workflows. Measured GPU service offers a more defensible allocation basis, although the paper does not evaluate a production billing system.
Where the evidence stops
The physical results come from one production platform, twenty workflows, NVIDIA H800 GPUs, and NVSHMEM. Production-trace request counts and geographic scope are not reported. The 256-GPU result is simulated, baseline coverage is not exhaustive, and no public artifact repository is identified in the source record.
Video generation is outside scope. Operators should validate three conditions before adopting the design logic: meaningful component reuse, persistent memory slack during compute-heavy inference, and sufficient control over the accelerator communication stack.
The serverless unit is becoming smaller than the application
ServerlessT2I’s strongest idea is architectural. A customized generative workflow becomes easier to scale after the platform can see its model stages, dependencies, memory demands, and service costs.
That shifts the deployment decision from “How many copies of this workflow?” to “Which stages need capacity, what can remain resident, and how should measured GPU service be divided?” The paper provides substantial evidence for that shift in bursty, reusable T2I workloads. Portability across accelerators, communication layers, and production demand remains the next test.
Cognaptus: Automate the Present, Incubate the Future.