Registry / llm-agents / tiktoken-rs-node

tiktoken-rs-node

library1.0.6jsnpmunverified

tiktoken-rs-node provides performant NAPI Rust bindings for the `tiktoken-rs` library, which implements OpenAI's `tiktoken` BPE tokenizer for encoding and decoding text to and from token integers. As of version 1.0.6, it offers a zero-copy encode mechanism and avoids WebAssembly (WASM), differentiating it from other Node.js `tiktoken` implementations that often rely on WASM. The package is actively maintained with frequent patch releases, indicating ongoing support for its current stable API. Its primary use case is for applications requiring efficient tokenization of text for large language models, particularly in Node.js environments where high performance and minimal overhead are critical. It supports a range of encodings including `cl100k_base`, `gpt2`, and `o200k_base`.

llm-agents ai-ml

npm install tiktoken-rs-node

github ↗package ↗

INSTALL

IMPORT

SIG · TIKTOKEN-RS-NODE

tiktoken-rs-node

llm-agentsjavascriptv1.0.6

Install

—

Import

—

Disk

—

Pass rate

0/ 6

Env Coverage0 / 6

glibc

18–22

musl

18–22

Install & Compatibility

Where this runs

tested against v? · npm install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

node 18–226 runs

build_error

glibc

node 18–226 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

getEncoding

✓ import { getEncoding } from 'tiktoken-rs-node';

✗ const getEncoding = require('tiktoken-rs-node');

The library primarily uses named exports. While CommonJS `require` works, ESM `import` is the recommended pattern, especially when using TypeScript.

Tiktoken

✓ import type { Tiktoken } from 'tiktoken-rs-node';

✗ import { Tiktoken } from 'tiktoken-rs-node';

The `Tiktoken` class itself is not directly exported for instantiation. `getEncoding()` returns an instance of `Tiktoken`. Import its type for strong TypeScript typing.

CommonJS

✓ const { getEncoding } = require('tiktoken-rs-node');

For CommonJS environments, destructuring `require` is the correct way to access `getEncoding`.

This quickstart demonstrates how to load an `tiktoken` encoding, tokenize a string, and then decode the tokens back into a string using the `tiktoken-rs-node` library.

import { getEncoding } from 'tiktoken-rs-node'; // Load the encoding once per application startup for optimal performance. // Supported encodings: 'o200k_base', 'cl100k_base', 'p50k_edit', 'p50k_base', 'r50k_base', 'gpt2' const encodingName = 'cl100k_base'; const encoding = getEncoding(encodingName); const textToEncode = "This is a sample sentence for tokenization using tiktoken-rs-node."; // Encode the text into an array of token integers const tokens = encoding.encode(textToEncode); console.log(`Original text: "${textToEncode}"`); console.log(`Encoded tokens (${encodingName}): [${tokens.join(', ')}]`); console.log(`Number of tokens: ${tokens.length}`); // Decode the tokens back into a string const decodedString = encoding.decode(tokens); console.log(`Decoded string: "${decodedString}"`); // Example with a different encoding const gpt2Encoding = getEncoding('gpt2'); const gpt2Tokens = gpt2Encoding.encode("Hello, world!"); console.log(`GPT-2 Encoded tokens: [${gpt2Tokens.join(', ')}]`);

Debug

Known issues

gotchaCalling `getEncoding` multiple times with the same encoding name can lead to unnecessary overhead. For long-running processes, it is highly recommended to call `getEncoding` only once per-startup and reuse the returned `Tiktoken` instance.

fix

Store the result of `getEncoding` in a module-level or application-level variable and reuse it across your application.

affects: >=1.0.0

breakingAs a Node.js NAPI binding, `tiktoken-rs-node` includes prebuilt binaries for common platforms. If your deployment environment (e.g., specific Docker image, serverless function, or CPU architecture) is not supported by the prebuilt binaries, you may encounter installation failures requiring local compilation, which needs a Rust toolchain.

fix

Check the project's GitHub releases for supported platforms. If prebuilt binaries are unavailable, ensure a Rust toolchain is installed in your build environment. Alternatively, consider using `npm install --build-from-source` if available.

affects: >=1.0.0

gotchaUsing an unsupported encoding name with `getEncoding` will result in a runtime error. Ensure you use one of the officially supported encoding names (e.g., 'cl100k_base', 'gpt2', 'o200k_base').

fix

Refer to the `tiktoken-rs-node` documentation or `tiktoken-rs` project for a list of valid encoding names.

affects: >=1.0.0

Errors

Common errors & fixes

Error: Cannot find module 'tiktoken-rs-node' or its corresponding type declarations.

The package either failed to install correctly (e.g., prebuilt binaries missing, build from source failed) or the import path is incorrect.

fix

Run `npm install tiktoken-rs-node` again. Check the installation logs for errors related to NAPI or Rust compilation. If running in a specific environment, verify Rust toolchain presence or ensure prebuilt binaries exist for your platform. Verify the import statement `import { getEncoding } from 'tiktoken-rs-node';`.

TypeError: encoding.encode is not a function

The `encoding` object returned by `getEncoding` is not the expected `Tiktoken` instance, or `getEncoding` itself failed to return a valid object. This might happen if `getEncoding` was called with an invalid name.

fix

Ensure `getEncoding` was called with a valid string encoding name (e.g., 'cl100k_base'). Add error handling around `getEncoding` calls. Verify `encoding` is not `undefined` or `null` before calling methods.

Upgrade

Version history

1.0.6latest on npm

Audit

Dependencies

No dependency data recorded yet.

Agent activity

30 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/tmb/tiktoken-rs-node ↗

packagehttps://www.npmjs.com/package/tiktoken-rs-node ↗