About Tokenizer
Tokenizer provides TypeScript and C# implementations of byte-pair encoding tokenization for language models. Tokenization splits text into smaller units used by a model, and matching the correct tokenizer helps estimate or prepare text in the form a model expects. The supplied description identifies compatibility with a particular family of large language models and says the implementation is based on an open-source Rust tokenizer. This makes the library useful to developers who need tokenization from TypeScript or C# code. It is a software component rather than an end-user chat application, and exact token counts depend on the chosen model encoding. The tokenizer is implemented in both listed languages. The tokenizer has implementations in TypeScript and C#.
Categories
Platforms
Website & Downloads
Trusted package or store listing
Related software
- .NET Reactor — .NET Reactor is a code-protection and software-licensing system for applications built on the .NET Framework.
- 010 Editor — 010 Editor is a text and hexadecimal editor with Binary Templates technology.
- 10x — 10x is a Windows code editor designed for large C++ projects.
- 8051 Disassembler — 8051 Disassembler converts Intel HEX files for the 8051 microcontroller family into readable assembly listings.
- ACS — ACS, or Admin Config Service, is a command-line interface for managing configuration and administrative tasks in a cloud platform.
- APK Editor Studio — APK Editor Studio is an APK editor and reverse-engineering tool for desktop computers.
