js-tiktoken

library1.0.21jsnpmunverified

js-tiktoken is a pure JavaScript port of OpenAI's tiktoken library, providing a BPE (Byte Pair Encoding) tokenizer primarily for use with OpenAI's models. Currently at version 1.0.21, the library undergoes frequent patch releases, mainly to incorporate new OpenAI models and their corresponding tokenizer configurations. Its key differentiators include being a pure JavaScript implementation, making it suitable for web browsers, edge environments, and Node.js applications where Python dependencies are not feasible. It also offers a "lite" mode, allowing developers to load only specific encoding ranks to significantly reduce bundle size, or to dynamically fetch encoding data from a CDN, addressing concerns about the full library's potentially large footprint.

llm-agents ai-ml

npm install js-tiktoken

github ↗package ↗

INSTALL

IMPORT

SIG · JS-TIKTOKEN

js-tiktoken

llm-agentsjavascriptv1.0.21

Install

—

Import

—

Disk

—

Pass rate

0/ 6

Env Coverage0 / 6

glibc

18–22

musl

18–22

Install & Compatibility

Where this runs

tested against v? · npm install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

node 18–226 runs

build_error

glibc

node 18–226 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

getEncoding

✓ import { getEncoding } from 'js-tiktoken';

✗ const getEncoding = require('js-tiktoken').getEncoding;

Primarily an ESM-first package. For CommonJS, named exports like `getEncoding` are properties of the `require` result, but native ESM is recommended.

encodingForModel

✓ import { encodingForModel } from 'js-tiktoken';

✗ const { encodingForModel } = require('js-tiktoken');

A named export for retrieving an encoding based on a model name. Requires correct ESM import syntax for optimal usage.

Tiktoken

✓ import { Tiktoken } from 'js-tiktoken/lite';

✗ import { Tiktoken } from 'js-tiktoken';

The `Tiktoken` class for the bundle-size-optimized 'lite' version must be imported from the `js-tiktoken/lite` subpath. Importing from the main package will not expose this specific class for the lite usage pattern.

Demonstrates how to obtain an encoding using both `getEncoding` for a specific scheme and `encodingForModel` for a model name, then encode and decode text, verifying the round trip. Includes error handling for unknown models.

import assert from 'node:assert'; import { getEncoding, encodingForModel } from 'js-tiktoken'; // Basic usage: Get an encoding directly const enc = getEncoding('gpt2'); const encodedTokens = enc.encode('hello world'); console.log(`'gpt2' tokens for 'hello world': ${encodedTokens}`); assert(enc.decode(encodedTokens) === 'hello world'); // Model-specific usage: Get encoding for a known model const modelName = 'gpt-4'; // Or 'gpt-3.5-turbo', 'text-embedding-ada-002', etc. try { const modelEnc = encodingForModel(modelName); const text = 'This is an example sentence for GPT-4 tokenization.'; const tokens = modelEnc.encode(text); console.log(`\n'${modelName}' tokens: ${tokens.length}, tokens array: [${tokens.slice(0, 5)}..., ${tokens.slice(-5)}]`); const decoded = modelEnc.decode(tokens); console.log(`'${modelName}' decoded: ${decoded}`); } catch (error) { console.error(`\nError getting encoding for model '${modelName}':`, error.message); }

Debug

Known issues

gotchaImporting the main `js-tiktoken` package directly (e.g., `import { getEncoding } from 'js-tiktoken';`) will bundle *all* OpenAI tokenizer data, which can significantly increase your application's bundle size, especially for web environments.

fix

For web applications or environments with strict bundle size limits, use the `js-tiktoken/lite` import path and load encoding ranks manually from `js-tiktoken/ranks/...` or dynamically fetch them from a CDN. Example: `import { Tiktoken } from 'js-tiktoken/lite'; import o200k_base from 'js-tiktoken/ranks/o200k_base'; const enc = new Tiktoken(o200k_base);`

affects: >=1.0.0

gotcha`js-tiktoken` is built as an ESM-first package. While CommonJS `require` might partially work for some exports through transpilation or Node.js interoperability, it is not the primary pattern, and can lead to unexpected behavior or larger bundle sizes in certain setups.

fix

Migrate projects to use native ESM imports (`import ... from '...'`) for optimal compatibility, tree-shaking benefits, and future-proofing. If strictly bound to CommonJS, consider dynamic `import()` or ensure your build setup correctly handles ESM-to-CJS conversion.

affects: >=1.0.0

gotchaModel names used with `encodingForModel` are frequently updated and can be case-sensitive. Using an unrecognized or incorrectly cased model name will result in an `Error: Unknown encoding`.

fix

Always refer to the official OpenAI documentation or `js-tiktoken`'s source for the exact and up-to-date list of supported model names. Implement error handling (e.g., `try...catch`) around calls to `encodingForModel` to gracefully manage unknown model names.

affects: >=1.0.0

Errors

Common errors & fixes

TypeError: Cannot find module 'js-tiktoken/ranks/o200k_base'

You are likely using the `js-tiktoken/lite` import strategy but your bundler (e.g., Webpack, Rollup, Vite) is not configured to correctly handle direct imports of the raw JSON rank data files from `js-tiktoken/ranks/`.

fix

Ensure your build tool is configured to process and include static JSON files. For example, in Webpack, you might need a `json-loader`. Alternatively, fetch the rank data dynamically from a CDN as outlined in the `js-tiktoken/lite` documentation example.

TypeError: getEncoding is not a function

This error typically occurs when trying to access a named export like `getEncoding` directly from a CommonJS `require` call in a package that is primarily ESM, or when destructuring is applied incorrectly.

fix

Use native ESM imports: `import { getEncoding } from 'js-tiktoken';`. If you must use CommonJS, ensure you are correctly accessing the named export, potentially via `const { getEncoding } = require('js-tiktoken');` though full ESM compatibility is recommended.

Error: Unknown encoding 'some-unrecognized-model-name'

The model name provided to `getEncoding` or `encodingForModel` does not match any of the currently supported models or encoding schemes within the `js-tiktoken` library.

fix

Verify the exact spelling and casing of the model name. Consult the `js-tiktoken` documentation or the underlying `tiktoken` library's list of supported models, as these are frequently updated with new OpenAI releases. Consider adding a `try...catch` block to handle cases where a model name might become deprecated or is not yet supported.

Upgrade

Version history

1.0.21latest on npm

Audit

Dependencies

No dependency data recorded yet.

Agent activity

23 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/dqbd/tiktoken ↗

packagehttps://www.npmjs.com/package/js-tiktoken ↗