Inference Frameworks

ONNX Runtime

microsoft

Cross-platform inference engine for ONNX models — graph-level optimization with pluggable hardware execution providers

Editorial compatibility: these components can be used together in representative deployments. That's not a requirement or a validated implementation. For real deployments, see the sourced reference architectures.

Compatible from earlier layers

RAG
Embeddings

Compatible with later layers

Triton Inference Server
NVIDIA NIM
LiteLLM