LLM Serving & Inference (vLLM & Ollama)

Introduction

Training a Large Language Model (LLM) is incredibly difficult, but what happens after it is trained? When millions of users send queries to ChatGPT every minute, the servers must generate text instantly. This phase is called Inference or Serving.

Naive inference—just running a standard PyTorch model.generate() loop in production—is notoriously slow, inefficient, and expensive. The bottleneck in LLM generation is not computing power (TFLOPS); it is Memory Bandwidth. Moving massive weight matrices from the GPU memory to the GPU processors for every single word generated takes time.

To serve LLMs to real users, we need specialized inference engines. Today, the two most important tools in this ecosystem are vLLM (the industry standard for high-throughput cloud deployment) and Ollama (the standard for running LLMs locally on personal laptops). Understanding how these engines optimize the inference process is critical for any AI engineer.

Core Concepts Explained Intuitively

1. The KV Cache Bottleneck

LLMs generate text auto-regressively: one word at a time. To generate the 100th word, the model must pay attention (Self-Attention) to the previous 99 words. Instead of recalculating the Keys (K) and Values (V) for the previous 99 words every single time, inference engines save them in a KV Cache.

However, in a server handling hundreds of users, the KV Cache grows massively and dynamically. In naive PyTorch, you must pre-allocate a huge contiguous block of memory for the maximum possible sequence length for every user. This leads to massive memory fragmentation and waste (up to 80% of GPU VRAM is wasted by pre-allocation).

2. vLLM and PagedAttention

vLLM solved the KV Cache problem by borrowing a concept from classic Operating Systems: Virtual Memory and Paging. Instead of allocating a massive contiguous block of memory for a user's KV cache, vLLM's PagedAttention algorithm breaks the cache into small, fixed-size blocks (e.g., blocks of 16 tokens).

These blocks are stored non-contiguously in the GPU memory. vLLM maintains a block table that maps logical tokens to physical blocks. This eliminates fragmentation and allows the server to batch vastly more users together, increasing throughput (tokens generated per second) by up to 400% compared to standard HuggingFace architectures.

3. Continuous Batching

Standard batching forces the server to wait until all requests in a batch are finished before starting a new one. If User A asks for 10 words and User B asks for 1000 words, User A's slot in the GPU sits idle while User B finishes. Continuous Batching (used by vLLM) evaluates requests dynamically at the iteration level. The millisecond User A finishes, their slot is immediately filled by User C's new request.

4. Ollama and Local Execution

While vLLM is designed for maximum throughput in the cloud, Ollama is designed for extreme ease-of-use on local hardware (Mac, Windows, Linux). It abstracts away all the complexities of Python, PyTorch, and CUDA. Under the hood, it uses llama.cpp, a C/C++ engine heavily optimized for running highly quantized (4-bit or 8-bit) models on consumer CPUs and Apple Silicon (M1/M2/M3 chips).

Code Examples

1. Beginner Code Example: Local Inference with Ollama

Ollama is not a Python library; it is a background service. However, it provides a seamless REST API and a Python client.

First, in your terminal, you would run: ollama run llama3 (This downloads and starts serving the model).

Then, you can interact with it using Python:

# pip install ollama
import ollama

print("Sending request to local Ollama server...")

# Stream the response as it generates (like ChatGPT UI)
response = ollama.chat(model='llama3', messages=[
  {
    'role': 'user',
    'content': 'Explain the concept of gravity to a 5 year old.',
  },
], stream=True)

for chunk in response:
    # Print each chunk of text as it is generated by the local GPU/CPU
    print(chunk['message']['content'], end='', flush=True)
    
print("\n\nDone!")

2. Intermediate Code Example: High-Throughput Serving with vLLM

If you have a server with a powerful GPU (like an A100), you use vLLM to process thousands of prompts instantly using PagedAttention.

# pip install vllm
from vllm import LLM, SamplingParams

# 1. Initialize the vLLM engine
# This loads the model and pre-allocates the PagedAttention KV Cache
print("Initializing vLLM engine...")
llm = LLM(model="facebook/opt-125m") # Using a tiny model for demonstration

# 2. Define Prompts (Imagine these came from 4 different users simultaneously)
prompts = [
    "The capital of France is",
    "The future of AI is",
    "To bake a chocolate cake, you need",
    "A summary of the movie The Matrix:"
]

# 3. Define Sampling Parameters (Temperature, Max Tokens, etc.)
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=50)

# 4. Generate! (vLLM automatically applies Continuous Batching here)
print("Generating responses...")
outputs = llm.generate(prompts, sampling_params)

# 5. Print the results
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"\nPrompt: {prompt!r}")
    print(f"Generated: {generated_text!r}")

3. Advanced Code Example: Spinning up an OpenAI-Compatible API with vLLM

vLLM's true power is that it can act as a drop-in replacement for OpenAI's API. You can host an open-source model, but all your existing software that expects to talk to api.openai.com will still work.

You don't write this in Python; you run it via the CLI on your server:

# Start the vLLM API server on port 8000
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3-8b-Instruct \
    --tensor-parallel-size 2 \
    --max-model-len 4096

Now, any client can query your GPU exactly as if it were querying OpenAI:

from openai import OpenAI

# Point the OpenAI client to YOUR local vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

completion = client.chat.completions.create(
    model="meta-llama/Llama-3-8b-Instruct",
    messages=[{"role": "user", "content": "Write a python script to reverse a string."}],
)
print(completion.choices[0].message.content)

Common Mistakes

1. Using HuggingFace pipeline for Production APIs:

   • Mistake: Wrapping a standard transformers model in a FastAPI endpoint. It will process one request at a time, cause massive queues, and crash with Out of Memory errors during peak traffic.
   • Fix: Always use a dedicated inference engine like vLLM, TGI (Text Generation Inference), or TensorRT-LLM for server deployments.

2. Ignoring the max_model_len Parameter in vLLM:

   • Mistake: Booting up a model without checking the max sequence length. vLLM tries to allocate KV Cache blocks for the maximum possible length (e.g., Llama-3's 8,000 tokens). If you only have 24GB of VRAM, vLLM might crash on startup because it can't allocate enough blocks.
   • Fix: If you only intend to generate short chat responses, limit the --max-model-len to 2048 to free up VRAM for more concurrent users.

3. Confusing Throughput with Latency:

   • Mistake: Optimizing a server for high batch size (vLLM) when building a voice-assistant app that requires instant response time.
   • Fix: vLLM is optimized for Throughput (tokens per second across all users). High batch sizes can slightly increase the Time to First Token (TTFT) (Latency) for an individual user. If you are building real-time voice AI, you might need different optimizations (like smaller batch sizes or faster hardware) over pure throughput.

Interview Q&A

Q1: What is the KV Cache in LLM Inference? A1: Because LLMs generate text auto-regressively, computing the next token requires the Key and Value matrices of all previous tokens in the sequence. Recomputing these from scratch every step is $O(N^2)$ complexity. The KV cache stores these matrices in memory so they only need to be computed once, changing the complexity to $O(N)$, massively speeding up generation.

Q2: How does vLLM's PagedAttention solve the memory fragmentation problem? A2: In standard systems, KV caches require contiguous memory allocation based on the maximum possible sequence length, leading to massive internal fragmentation (wasted space) because most sequences are shorter than the maximum. PagedAttention divides the KV cache into small, fixed-size blocks (pages). Tokens are mapped to these non-contiguous physical blocks dynamically via a block table, virtually eliminating wasted space and allowing the server to batch more users.

Q3: What is Continuous Batching (or Iteration-Level Batching)? A3: Standard batching processes a group of requests until the longest request finishes, leaving GPUs idle when shorter requests end early. Continuous batching evaluates requests at every token generation step. As soon as one request finishes, it is ejected from the batch, and a new incoming request is instantly injected in its place, maximizing GPU utilization.

Q4: Why does Ollama run so fast on Apple MacBooks? A4: Ollama utilizes llama.cpp under the hood. Apple Silicon chips (M-series) have a Unified Memory Architecture, meaning the CPU and GPU share the exact same RAM. While PC users are bottlenecked by moving data over the PCIe bus to the GPU, MacBooks can stream data directly from high-speed RAM. Additionally, Ollama uses advanced CPU vector instructions (like Apple AMX) to accelerate math.

Q5: What is Tensor Parallelism? A5: When a model is too large to fit on a single GPU (like a 70B parameter model needing 140GB of VRAM), Tensor Parallelism splits the actual matrix multiplication operations across multiple GPUs. During a single forward pass, GPU 1 calculates half of a linear layer, GPU 2 calculates the other half, and they communicate the results instantly via high-speed connections like NVLink.

Practice Problems

1. Problem: Install Ollama on your local machine, run the phi3 model (Microsoft's highly capable tiny model), and use the Python client to write a script that summarizes a hard-coded news article.
2. Problem: Explain in your own words the difference between Time to First Token (TTFT) and Generation Tokens Per Second. Which metric is more important for a Chatbot UI?
3. Problem: If an LLM has a hidden dimension of 4096, 32 layers, and uses 16-bit precision (2 bytes per number), calculate the rough memory size of the KV cache for a single token. (Hint: You need a Key vector and a Value vector for every layer).

Summary

• Inference is the process of generating text from a trained model. It is bottlenecked by Memory Bandwidth, not compute.
• The KV Cache saves previous token calculations to speed up generation, but consumes massive amounts of RAM.
• vLLM uses PagedAttention to eliminate memory fragmentation, allowing massive increases in user throughput on cloud servers.
• Continuous Batching ensures the GPU is never sitting idle waiting for a long sequence to finish.
• Ollama makes local LLM execution trivial by wrapping optimized C++ inference engines and handling quantization seamlessly.