Search

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Blazingly fast LLM inference.

Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

Inference for text-embeddings in Rust, HFOIL Licence.

Inference for text-embeddings in Python

Efficient Triton Kernels for LLM Training.

A distributed implementation of llama.cpp that lets you run 70B-level LLMs on your everyday devices.

Easily deploy any LLM on a VM with minimal configuration, using Ansible.

Comprehensive set of tools for working with local LLMs for various tasks.

Use ChatGPT On Wechat via wechaty

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.

Build your own conversational search engine using less than 500 lines of code by [LeptonAI](https://github.com/leptonai).

Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.

Playground for devs to finetune & deploy LLMs

Simple API for deploying any RAG or LLM that you want adding plugins.

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference

An open-source GPU cluster manager for running LLMs

- A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)