High-performance library for tokenizing text.
Cross-referenced across 55 tracked directories
#3754
Popularity Rank
1 / 55
Listed In
Emerging
Adoption Stage
2/3/2025
Created
2
GitHub Stars
Run an AI-powered security scan to analyze this package's source code for vulnerabilities, prompt injection vectors, data exfiltration risks, and behavior mismatches.
Scans fetch actual source code from the GitHub repository, not just the README.
nav2510
This package reviews the changes in each PR, generates summaries per file, adds relevant comments, suggestions, and highlights areas that need fixing.
...moreyazin
Validate and verify Quranic verses in LLM-generated text with 100% accuracy
brankrts
A TypeScript library for parsing and processing structured data from LLM streaming responses with custom tag definitions and event-driven architecture
...moresceuick
| [NPM Package](https://www.npmjs.com/package/@mlc-ai/web-tokenizers) | [WebLLM](https://github.com/mlc-ai/web-llm) |
1
Open Issues
9/8/2025
Last Commit
Recently added to the ecosystem