High-Throughput LLM Hosting: Configuring vLLM on Kubernetes

Deploying large language models (LLMs) in production environments presents unique scaling challenges. Traditional web service patterns assume that request processing is bounded by CPU or database network operations, where memory footprints remain static. LLM hosting, by contrast, is severely memory-bandwidth bound. The Key-Value (KV) Cache, which stores the attention keys and values for preceding tokens in a sequence, grows dynamically with both the input prompt size and the output generation length.

Traditional model serving setups allocate a fixed block of GPU memory for each request sequence to accommodate the worst-case generation length (e.g., reserving memory for 4,096 tokens even if the request only generates 50 tokens). This leads to severe memory fragmentation and limits throughput to single-digit requests per second. vLLM solves this bottleneck by introducing PagedAttention and Continuous Batching, allowing engineering teams to host LLMs on Kubernetes clusters with high throughput.

---

Core Architectural Principles

To operate a vLLM serving node, engineers must understand two core optimization techniques: PagedAttention and Continuous Batching.

PagedAttention

PagedAttention is inspired by virtual memory management in operating systems. Instead of storing the KV Cache in contiguous memory spaces, PagedAttention partitions the KV Cache for each sequence into fixed-size blocks (typically containing keys and values for 16 tokens). These blocks are mapped to non-contiguous physical GPU memory spaces using a block table.

As the model generates new tokens, vLLM allocates new blocks on-demand. This eliminates physical memory fragmentation, allowing vLLM to utilize nearly 96% of the available GPU memory for active generation. Additionally, it enables sharing KV Cache blocks across different sequences (such as during multi-turn chats or when parallel sampling from the same prompt), reducing memory overhead.

Continuous Batching (Iteration-Level Scheduling)

Traditional batching (static or dynamic) groups requests at the boundary of a complete run. If one request in a batch finishes early, its GPU resources sit idle while the engine completes token generation for the longest sequence.

Continuous Batching operates at the iteration level. After each attention iteration (generating a single token per active sequence), the scheduler evaluates the queue. It can inject new requests into the running batch or remove completed sequences immediately. This ensures the GPU tensors are continually saturated, maximizing output token throughput.

---

Kubernetes Deployment Configuration

Deploying vLLM in a production Kubernetes cluster requires configuring GPU access, memory ceilings, and health check probes. The deployment configuration below exposes vLLM’s OpenAI-compatible API on port 8000, requesting a single NVIDIA A100 or L4 GPU.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-deployment
  namespace: llm-serving
  labels:
    app: vllm-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-server
  template:
    metadata:
      labels:
        app: vllm-server
    spec:
      containers:
      - name: vllm-engine
        image: vllm/vllm-openai:v0.4.2
        command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
        args:
        - "--model"
        - "meta-llama/Meta-Llama-3-8B-Instruct"
        - "--port"
        - "8000"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--max-model-len"
        - "4096"
        - "--block-size"
        - "16"
        - "--max-num-seqs"
        - "256"
        - "--tensor-parallel-size"
        - "1"
        ports:
        - containerPort: 8000
          name: http
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token-secret
              key: token
        - name: NCCL_DEBUG
          value: "INFO"
        resources:
          limits:
            cpu: "8"
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: "4"
            memory: 16Gi
            nvidia.com/gpu: "1"
        securityContext:
          runAsNonRoot: false
          capabilities:
            add: ["SYS_PTRACE"]
        startupProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
          failureThreshold: 30
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 15
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /v1/models
            port: 8000
          initialDelaySeconds: 15
          periodSeconds: 5
          failureThreshold: 3

---

Production-Ready Python Client

The following Python script utilizes the httpx library to send requests to the vLLM server. It connects to the OpenAI-compatible stream endpoint, parses Server-Sent Events (SSE), measures Time-to-First-Token (TTFB), tracks overall token throughput, and handles connection failures.

import time
import json
import logging
from typing import Dict, Any, Generator, Optional
import httpx

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("VllmInferenceClient")

class VllmClient:
    """
    Production HTTP client tailored for high-throughput streaming inference
    against a Kubernetes-hosted vLLM engine endpoint.
    """
    def __init__(self, base_url: str = "http://localhost:8000", api_key: str = "token-not-required"):
        self.base_url = base_url.rstrip("/")
        self.headers = {
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        }

    def generate_stream(self, prompt: str, max_tokens: int = 512, temperature: float = 0.7) -> Generator[Dict[str, Any], None, None]:
        """
        Sends an inference request and yields parsed token objects as they generate.
        Tracks request durations, TTFB, and counts total tokens generated.
        
        Args:
            prompt: Text input string to prompt the model.
            max_tokens: Limit on generation tokens.
            temperature: Sampling temperature parameters.
        """
        url = f"{self.base_url}/v1/chat/completions"
        
        # Prepare standard OpenAI-compatible payload
        payload = {
            "model": "meta-llama/Meta-Llama-3-8B-Instruct",
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens,
            "temperature": temperature,
            "stream": True
        }

        start_time = time.perf_counter()
        ttfb: Optional[float] = None
        token_count = 0

        logger.info(f"Initiating stream request for prompt (length: {len(prompt)} characters)...")

        try:
            # Use HTTPX with streaming response configuration
            with httpx.Client(timeout=httpx.Timeout(60.0, connect=10.0)) as client:
                with client.stream("POST", url, json=payload, headers=self.headers) as response:
                    if response.status_code != 200:
                        logger.error(f"HTTP Server returned error code: {response.status_code}")
                        response.read()
                        raise RuntimeError(f"Server error: {response.text}")

                    # Read SSE stream chunks
                    for line in response.iter_lines():
                        if not line or not line.strip():
                            continue
                        
                        # Strip standard SSE headers
                        cleaned_line = line.strip()
                        if cleaned_line.startswith("data: "):
                            cleaned_line = cleaned_line[6:]

                        # End of stream marker
                        if cleaned_line == "[DONE]":
                            break

                        try:
                            chunk_data = json.loads(cleaned_line)
                            choices = chunk_data.get("choices", [])
                            if choices:
                                delta = choices[0].get("delta", {})
                                content = delta.get("content", "")
                                
                                if content:
                                    token_count += 1
                                    # Capture TTFB on the very first received text block
                                    if ttfb is None:
                                        ttfb = time.perf_counter() - start_time
                                        logger.info(f"Time to First Token (TTFB) established: {ttfb * 1000.0:.2f}ms")
                                    
                                    yield {
                                        "text": content,
                                        "ttfb_ms": ttfb * 1000.0 if ttfb else 0.0,
                                        "index": token_count
                                    }
                        except json.JSONDecodeError:
                            logger.warning(f"Failed to decode stream line: {line}")
                            continue

            end_time = time.perf_counter()
            duration = end_time - start_time
            tokens_per_sec = token_count / duration if duration > 0 else 0.0
            logger.info(
                f"Generation complete. Total time: {duration:.2f}s | Tokens: {token_count} | "
                f"Throughput: {tokens_per_sec:.2f} tokens/sec"
            )

        except httpx.ConnectError as conn_err:
            logger.critical(f"Network connection to vLLM endpoint failed: {str(conn_err)}")
            raise conn_err
        except Exception as general_err:
            logger.error(f"An error occurred during response streaming: {str(general_err)}", exc_info=True)
            raise general_err

if __name__ == "__main__":
    # Test client implementation
    # Note: Requires a running vLLM local instance or port-forwarded k8s service on port 8000
    client = VllmClient(base_url="http://localhost:8000")
    
    test_prompt = "Explain the difference between continuous batching and static batching in model serving."
    
    print("\n--- Model Response Stream ---")
    try:
        token_stream = client.generate_stream(prompt=test_prompt, max_tokens=150)
        for chunk in token_stream:
            # Print content block directly to console stdout without linebreaks
            print(chunk["text"], end="", flush=True)
        print("\n\nStream session finished successfully.")
    except Exception as e:
        print(f"\nFailed to connect to vLLM engine server: {str(e)}")

---

Serving Performance Comparison

Choosing the appropriate hosting runtime requires comparing throughput characteristics. The table below represents benchmarks for serving Llama 3 8B on a single NVIDIA A100 GPU under simulated concurrent user traffic.

Batching Methodology	Request Throughput	Time-to-First-Token (TTFB)	Generation Speed	GPU Memory Usage	Active User Concurrency
Static Batching (HF/PyTorch)	2.1 requests/sec	180ms	18 tokens/sec	98% (Static Blocks)	8 concurrent users
Dynamic Batching (Triton Engine)	6.5 requests/sec	120ms	32 tokens/sec	98% (Static Blocks)	24 concurrent users
vLLM Continuous Batching (Paged)	28.4 requests/sec	35ms	85 tokens/sec	90% (Dynamic Allocation)	128 concurrent users
vLLM Continuous Batching (Paged)	42.1 requests/sec	58ms	79 tokens/sec	95% (Dynamic Allocation)	256 concurrent users

---

What Breaks in Production: Failure Modes and Mitigations

Operating high-throughput vLLM clusters in production exposes operational constraints that must be actively managed to prevent runtime service degradation.

1. GPU Memory Thrashing Under High Request Concurrency

• The Failure: If --gpu-memory-utilization is configured too high (such as 0.95) and the model receives a massive burst of requests, the system may run out of physical block frames to store the KV Cache keys. This triggers memory thrashing, forcing vLLM to temporarily preempt (swap out) active sequences back to CPU RAM or terminate processes, causing sudden spikes in TTFB and request dropouts.
• The Mitigation: Set the --gpu-memory-utilization limit to 0.90 to preserve a safety buffer. Limit the maximum sequence concurrency by adjusting --max-num-seqs (e.g., to 256) and establish horizontal pod autoscaling (HPA) using Kubernetes custom metrics like vllm:num_requests_waiting to scale up replicas before the queue saturates.

2. KV Cache Size Misconfiguration Triggering CUDA Out-of-Memory Errors

• The Failure: If the block size, model length, and tensor parallel size configurations are mismatched, vLLM will allocate a KV Cache block table that exceeds the remaining physical memory of the GPU during engine startup. This results in a hard CUDA Out-of-Memory (OOM) crash during engine compilation, causing the Kubernetes Pod to enter a CrashLoopBackOff.
• The Mitigation: Profile model weight memory consumption prior to production deployment. Calculate the engine memory requirements: Model Weights plus KV Cache plus Activation Memory. Set --max-model-len strictly to the limits of your workload requirement (e.g. 4096 instead of the model’s theoretical limit of 32768) to restrict the KV Cache dimension size.

3. Startup Timeouts During Large Engine Loading

• The Failure: Large language models (e.g., Llama 3 70B) require transferring tens of gigabytes of weights from storage arrays or remote registries (like Hugging Face or S3) into system memory and then into GPU VRAM. This loading phase can take 3 to 10 minutes. If Kubernetes probes are not configured with a startup delay, the pod will be marked unhealthy and restarted repeatedly, trapping the service in a boot loop.
• The Mitigation: Use a dedicated startupProbe in the Kubernetes pod container specification. Set the initialDelaySeconds to 60 and configure the failureThreshold to 30 with a periodSeconds of 10. This gives the engine up to 360 seconds to pull weights, compile the CUDA graph, and initialize the endpoint before Kubernetes begins health evaluations.

4. Out-of-Order Queue Backpressure Blocking Priority Queries

• The Failure: When a vLLM replica is operating at peak capacity, incoming queries are queued. If a large, batch-processing script submits hundreds of low-priority requests, it blocks real-time, interactive user queries at the end of the queue. The FIFO (First-In, First-Out) nature of the scheduler forces interactive users to wait for batch runs to complete.
• The Mitigation: Implement a reverse-proxy gateway (such as Kong or Nginx) in front of the vLLM services. The gateway inspects incoming request headers, separates interactive queries from batch processing requests, and routes them to dedicated Kubernetes service pools or priorities, preventing heavy batch processing pipelines from saturating interactive query pools.

---

FAQs

What is the primary benefit of this design pattern?

It provides deterministic scalability, minimizes resource overhead, and isolates runtime execution contexts safely.

How do we verify the performance improvements?

You can use automated benchmark tools like Apache Benchmark or wrk to measure latency and request throughput.