Inference Frameworks

vLLM

PagedAttention-based LLM serving engine with continuous batching, speculative decoding, and an OpenAI-compatible API

Editorial compatibility: these components can be used together in representative deployments. That's not a requirement or a validated implementation. For real deployments, see the sourced reference architectures.

Compatible from earlier layers

Text Generation
RAG
Embeddings
Code Completion
Tool Calling
Multi-Turn Agent Loop
Sub-Agent Fan-Out

Compatible with later layers

Triton Inference Server
Ray Serve
NVIDIA NIM
NVIDIA Dynamo
LiteLLM
Envoy AI Gateway