Rohan Paul
@rohanpaul_ai
New Google paper shows for small quantized LLMs on serverless CPUs, 55–70% of cold-start latency is just model loading.
i.e. the bottleneck is often moving model weights, not generating tokens.
The inference itself is less of the problem than getting the model into memory.
Giving the same model 8 GB of Cloud Run memory instead of 4 GB unlocks roughly 2× the CPU, cutting warm inference time almost in half.
