Search

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Blazingly fast LLM inference.

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

Inference for text-embeddings in Rust, HFOIL Licence.

Inference for text-embeddings in Python

A distributed implementation of llama.cpp that lets you run 70B-level LLMs on your everyday devices.

Easily deploy any LLM on a VM with minimal configuration, using Ansible.

Use ChatGPT On Wechat via wechaty

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.

Build your own conversational search engine using less than 500 lines of code by [LeptonAI](https://github.com/leptonai).

Playground for devs to finetune & deploy LLMs

Simple API for deploying any RAG or LLM that you want adding plugins.

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference

An open-source GPU cluster manager for running LLMs

- A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

First LLM Multi-agent framework.

An interactive chat project that leverages Ollama/OpenAI/MistralAI LLMs for rapid understanding and navigation of GitHub code repository or compressed file resources.