Tech • AI • Robotics
AI Infrastructure Explained (GPUs, vLLM, and LLM-D)
8/10TL;DR
AI infrastructure has become a major engineering priority as hyperscalers pour more than $700 billion into systems for model serving, pushing demand for expertise in GPUs, model servers, and Kubernetes-based orchestration.
KEY POINTS
Spending surge reshapes infrastructure demand
Amazon, Google, Microsoft, and Meta are projected to spend over $700 billion on AI infrastructure this year, roughly four times their level of four years ago and nearly 80% higher than the previous year. McKinsey expects close to $7 trillion to flow into data centers by 2030, making the buildout of compute, memory, networking, and operations a central industry priority.
Models are software structures plus learned weights
A language model is fundamentally a fixed computational structure plus a massive set of learned numerical weights. In modern systems that structure is typically the transformer, while the model’s capabilities come from the weights produced during training. Those weights are stored as files that can range from about 2 GB for small models to 140 GB for a 70 billion-parameter model, with the largest systems reaching several hundred gigabytes.
Why laptops struggle and GPUs dominate
Running a model requires billions of small multiplications that can be executed in parallel, which makes GPUs far better suited than CPUs. A CPU may deliver roughly 10 trillion operations per second, while a GPU can reach around 1,000 trillion, or about 100 times more math throughput. The challenge is not only compute but feeding the processor fast enough with model weights.
VRAM and bandwidth are decisive constraints
GPUs solve the data-feeding problem with on-card VRAM, which sits close to the compute cores and delivers bandwidth in terabytes per second rather than the far slower system-memory path. But that speed comes with limited capacity. A card such as an Nvidia T4 has about 16 GB of VRAM, while data-center parts such as the A100, H100, H200, and B200 offer far higher compute and memory, up to 192 GB of VRAM and about 8 TBps bandwidth on the B200.
Model serving turns a script into an always-on service
Frameworks such as PyTorch load model weights onto GPUs, while serving layers such as vLLM expose them through a web API compatible with the OpenAI API format. That makes it possible to swap a hosted model endpoint for a self-run server without changing client code. The drawback is cost: each server keeps a full copy of the model in GPU memory, making startup slow and replication expensive.
Token generation explains latency
Language models work in tokens, not full words, and generate output one token at a time. That creates two performance phases: prefill, when the full prompt is processed in one burst, and decode, when the answer is produced token by token. The main latency metrics are TTFT or time to first token, and TPOT or time per output token.
Caching is essential to performance and cost
During prefill, the model stores reusable intermediate work in KV cache, avoiding a full recomputation for every generated token. A further optimization, prefix caching, retains work tied to repeated prompt prefixes, which cuts both latency and cost in chat-style systems and shared assistant workloads. Some commercial APIs already discount cached input tokens to about one-tenth the price of uncached tokens.
Batching boosts throughput, but memory sets the limit
Serving multiple users efficiently depends on batching, which lets one read of the model produce tokens for many requests at once. That sharply raises total throughput, but each user also needs KV cache in VRAM. In practice, GPU memory often becomes the real bottleneck before raw compute does, forcing operators to queue users or add more hardware.
Large models must be split across GPUs
When a model is too large for one card, operators use sharding across multiple GPUs. Inside one machine, fast links such as NVLink allow frequent communication and finer-grained splits; across machines, slower networks favor coarser layer-by-layer partitioning. The result is one logical server spread over several GPUs or several machines.
Traditional load balancing fails for LLMs
Standard round-robin balancing wastes performance because it ignores where each user’s cache resides and assumes requests are similar in size. In reality, one request may be a short question and another a 50-page summarization job. Sending follow-up messages to the wrong server forces expensive recomputation and can slow dozens of active users sharing that GPU.
LLM-D aims to route requests intelligently
LLM-D, backed by Red Hat, Google, IBM, and Nvidia, is designed as a smarter routing layer for model-serving fleets. It considers cached state, free memory, and queue depth before placing requests. Cache-aware routing can deliver about 3 times the throughput and cut first-response latency by roughly half, while separating prefill and decode onto different GPU pools can raise output by up to 70% more tokens per second on the same hardware.
Kubernetes is becoming the control plane
The orchestration layer for this infrastructure is increasingly Kubernetes, already used by about 66% of organizations hosting generative AI inference workloads. In this model, inference servers run as pods, routing sits behind a gateway and scheduler, and giant sharded models can be grouped with newer constructs such as LeaderWorkerSet. The result is a familiar operational pattern for platform teams, with standard restart, placement, and scaling behavior applied to AI workloads.
CONCLUSION
The core challenge in generative AI is no longer only model quality but the infrastructure required to run models quickly, cheaply, and at scale. As spending accelerates, expertise in GPU systems, serving software, and Kubernetes-native orchestration is becoming a critical part of the AI stack.
Explain this