lenML

claude tokenizer for NodeJS/Browser

Our library `@lenml/llama2-tokenizer` has been deprecated. We are excited to introduce our new library `@lenml/tokenizers` as its replacement, offering a broader set of features and an enhanced experience.

gemini tokenizer for NodeJS/Browser

gemma3 tokenizer for NodeJS/Browser

a lightweight no-dependency fork of transformers.js (only tokenizers)

gemma tokenizer for NodeJS/Browser

deepseek_v3 tokenizer for NodeJS/Browser

mistral_nemo tokenizer for NodeJS/Browser

deepseek_v4 tokenizer for NodeJS/Browser

qwen3 tokenizer for NodeJS/Browser

llama3 tokenizer for NodeJS/Browser

llama2 tokenizer for NodeJS/Browser

gemma2 tokenizer for NodeJS/Browser

gptoss tokenizer for NodeJS/Browser

llama3_1 tokenizer for NodeJS/Browser

gpt4o tokenizer for NodeJS/Browser

gpt35turbo tokenizer for NodeJS/Browser

gpt4 tokenizer for NodeJS/Browser

qwen2_5 tokenizer for NodeJS/Browser

chatglm3 tokenizer for NodeJS/Browser

gpt2 tokenizer for NodeJS/Browser

llama3_2 tokenizer for NodeJS/Browser

yi tokenizer for NodeJS/Browser