To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
Cross-referenced across 55 tracked directories
#3494
Popularity Rank
1 / 55
Listed In
Emerging
Adoption Stage
5/22/2024
Created
1,198
GitHub Stars
76
Forks
90
Open Issues
3/9/2026
Last Commit
Recently added to the ecosystem
Score: 100/100
0 dependency vulnerabilities found
vLLM Team
A high-throughput and memory-efficient inference and serving engine for LLMs
SkyPilot Team
Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.
...moreA high-throughput and low-latency inference and serving framework for LLMs and VLs