Skills

The API for building streaming tokenizers and lexers.

Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.

A simple tool to generate bert tokens and input features

Official AstroPay client for easy frontend integration to Tokenizer

Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.

A wide purpose tokenizer for node.js which looks like a stream

Wix Restaurants credit-cards tokenizer

String Tokenizer for Node.js using ICU's BreakIterators

A Vue 3 UI Kit for Tokenizer - Complete component library

A fast tokenizer/lexer for JavaScript

A regular expression (RexExp) stream tokenizer.

A fast tokenizer/lexer core that is well tested.

Extract a shader's name from its source

Javascript text tokenizer that is easy to use and compose

parse me!

finite state machine tokenizer

a tokenizer for remark to support details plugin in pymdown

JavaScript implementation of Japanese morphological analyzer

Javascript text tokenizer that is easy to use and compose

## Documentation [View Documentation](https://sandbox.fluidpay.com/docs/tokenizer/)

<!-- Do not edit directly, built using contentful-readme-generator. Content details in Build Information below. -->

Capacitor 8 plugin for Conekta card tokenization - create payment tokens from card details via direct API calls

BPE tokenizer used for the NovelAI frontend.

A lightweight CSS tokenizer and parser that produces a simple AST for custom CSS processing.