Skills

The Astropay Tokenizer SDK is a tool to tokenize and reveal sensitive payment information

NFA Tokenizer

A minimalistic, general purpose tokenizer generator

A simple, Twitter-aware tokenizer.

A lexical analyser and generator for typescript.

Vietnamese Tokenizer for JS to fill the gap that Intl.Segmentor doesn't combine Vietnamese syllables to words.

Takes a string and make tokens

A powerful and lightweight multilingual tokenizer library that provides natural language processing capabilities for multiple languages including English, Chinese, Japanese, and Korean.

Lightweight tokenizer and syntax highlighter.

Collection of NodeJS NLP Library for Bahasa Indonesia.

Tokenizer for english sentences

Lexer generator from RegExp spec

A tokenizer for Google-like search queries

Compile javascript into an AST to be consumed by other tools

NLTK TreebankTokenizer Ported from Python to JS

moo (ultra-fast tokenizer) plugin for nearley

A JS implementation of a tokenizer to split up source code based on tokens

Convert Chinese text to list of Chinese words

Simple string tokenizer for lexical scanning operations

render markdown link in react

Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!

Generator function that tokenizes string based on JSON format.

Preprocess glsl es (webgl) shader code: lint, minimize, obfuscate and/or optimize

A simple perfomant tokenizer