Busy Compute, Idle Memory: ServerlessT2I Rebuilds Image Serving Around Model DAGs
TL;DR for operators A serverless image platform must handle many bursty workflows that reuse some models and adapters but differ in computation, loading cost, and latency requirements. The usual assumption is that each customized workflow should scale as one indivisible GPU deployment. ServerlessT2I shows why that unit can be too coarse. In its production trace, median GPU compute utilization reached 100%, while memory use was only 16 GiB at the median and 35 GiB at the 95th percentile. Compute was saturated, but substantial memory remained available. Monolithic deployment strands that capacity because the platform cannot independently scale, place, or cache the reusable stages inside each workflow. ...