Skills

BPE tokenizer used for the NovelAI frontend.

A lightweight CSS tokenizer and parser that produces a simple AST for custom CSS processing.

A super fast diffing utility for HTML and Plaintext

Tokenizer inspired by Laravel's Sanctum

simple code like tokenizer

NPM credit card tokenizer

Split text into sentences with Sentence Boundary Detection (SBD).

A React Native supported JavaScript implementation of Japanese morphological analyzer

Tokenize CSS according to the CSS Syntax

render markdown code in react

React-based typeahead and typeahead-tokenizer

kuromoji neologd dictionary

A port of NLTK's Punkt sentence tokenizer to JS.

CardConnect Hosted iFrame Tokenizer & PCI Compliant CardSecure tokenization package to securely authorize & capture transactions with NativeStack & CardPointe on React.js.

Advanced tagging/tokenizing plugin for input fields with a focus on keyboard and copy-paste support.

PAYONE Commerce Platform Client JavaScript SDK

tiny tokenizer for simple parsing

A Bedrock module to provide auto-rotating tokenizers

Tokenizer for Vietnamese in Nodejs and Javascript

A Metalsmith plugin to render markdown files to HTML

A promise based streaming tokenizer

The Astropay Tokenizer SDK is a tool to tokenize and reveal sensitive payment information

NFA Tokenizer

A minimalistic, general purpose tokenizer generator