vLLM

A high-throughput and memory-efficient inference and serving engine for LLMs

vLLM CPU inference engine (AVX512 + VNNI + BF16 + AMX optimized)

vLLM CPU inference engine (AVX512 + VNNI + BF16 optimized)

vLLM CPU inference engine (AVX512 + VNNI optimized)

vLLM CPU inference engine (AVX512 optimized)

A high-throughput and memory-efficient inference and serving engine for LLMs

A high-throughput and memory-efficient inference and serving engine for LLMs

A framework for efficient model inference with omni-modality models

vLLM Semantic Router fleet simulator for capacity planning, SLO validation, and what-if analysis

LLM Katan - Lightweight LLM Server for Testing - Real tiny models with FastAPI and HuggingFace

A library for compressing large language models utilizing the latest techniques and research in the field for both training aware and post training techniques. The library is designed to be flexible and easy to use on top of PyTorch and HuggingFace Transformers, allowing for quick experimentation.

vLLM Ascend backend plugin

vLLM Semantic Router - Intelligent routing for Mixture-of-Models

A high-throughput and memory-efficient inference and serving engine for LLMs

AIBrix, the foundational building blocks for constructing your own GenAI inference infrastructure.

A high-throughput and memory-efficient inference and serving engine for LLMs

A high-throughput and memory-efficient inference and serving engine for LLMs

A high-throughput and memory-efficient inference and serving engine for LLMs

A high-throughput and memory-efficient inference and serving engine for LLMs