mistral.rs

LLM inference in C/C++.

Nvidia Framework for LLM Inference

A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.