js-tokens

library10.0.0jsnpmunverified

js-tokens is a JavaScript tokenizer that leverages regular expressions to parse JavaScript code into a stream of tokens. It is known for its tiny footprint, lenient parsing approach, and robustness, designed to 'never fail' even on malformed input, making it suitable for tools that need to process potentially incomplete or invalid JavaScript. The current stable version is 10.0.0. While not strictly spec-compliant in all edge cases, its 'almost spec-compliant' nature makes it highly practical for tasks like syntax highlighting, basic static analysis, or code transformation where full AST parsing is overkill. It supports the latest ECMAScript features up to ES2025 and also offers an option for JSX support. The project maintains a steady release cadence for new ECMAScript features and bug fixes, with major versions often introducing changes to module formats or API surface to align with ecosystem shifts.

serialization

npm install js-tokens

github ↗package ↗

INSTALL

IMPORT

SIG · JS-TOKENS

js-tokens

serializationjavascriptv10.0.0

Install

—

Import

—

Disk

—

Pass rate

0/ 6

Env Coverage0 / 6

glibc

18–22

musl

18–22

Install & Compatibility

Where this runs

tested against v? · npm install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

node 18–226 runs

build_error

glibc

node 18–226 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

jsTokens

✓ import jsTokens from 'js-tokens';

✗ const jsTokens = require('js-tokens');

Since v10.0.0, js-tokens is an ES Module. While Node.js 20.19.0+ supports `require(esm)`, it's generally recommended to use `import` for full compatibility and better tooling. For older CommonJS environments, dynamic `import()` or specific bundler configurations are necessary.

Token type

✓ import type { Token } from 'js-tokens';

✗ import { Token } from 'js-tokens';

js-tokens ships its own TypeScript types since v5+, making `@types/js-tokens` unnecessary. Ensure `esModuleInterop: true` in your `tsconfig.json` for seamless type resolution.

jsTokens (JSX option)

✓ import jsTokens from 'js-tokens'; jsTokens(jsString, { jsx: true });

The tokenizer supports an optional `jsx: true` flag to enable JSX tokenization, affecting how angle brackets and identifiers are parsed.

Demonstrates basic tokenization of a JavaScript string, extracting token values. Also shows enabling JSX support.

import jsTokens from 'js-tokens'; const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\t")'; const tokens = Array.from(jsTokens(jsString)); console.log(tokens.map(token => token.value).join('|')); // Expected output: JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|) // Example with JSX support const jsxString = 'const element = <div>Hello, {name}!</div>;'; const jsxTokens = Array.from(jsTokens(jsxString, { jsx: true })); console.log(jsxTokens.map(token => token.type + ':' + token.value).join(', '));

Debug

Known issues

breakingjs-tokens v10.0.0 transitioned to being an ES Module (ESM) package, setting `"type": "module"` in its `package.json`. This fundamentally changes how it should be imported.

fix

For CommonJS projects, use dynamic `import('js-tokens')` or ensure you are on Node.js 20.19.0 or later which supports `require(esm)`. For new projects, adopt `import` statements and ensure your project is configured for ESM.

affects: >=10.0.0

breakingIn v6.0.0, the main export changed from a direct regular expression to a generator function. Code relying on the raw regex pattern or its `matchToToken` helper will break.

fix

Update your code to call `jsTokens(string, options?)` and iterate over the returned generator function, rather than using a regex directly.

affects: >=6.0.0 <10.0.0

breakingVersion 4.0.0 introduced breaking changes including requiring Node.js 10+ (due to Unicode property escapes), aligning token names with the ECMAScript spec, and treating whitespace/line terminators as separate tokens.

fix

Ensure your environment runs Node.js 10 or newer. Review code that relies on specific token structures, names, or how whitespace was previously handled, and adjust to the updated specification-aligned token output.

affects: >=4.0.0 <6.0.0

gotchaThe tokenizer is described as 'almost spec-compliant' and takes 'a couple of shortcuts'. While it passes most `test262-parser-tests`, it has documented limitations (e.g., regarding `>>` and `>>>` operators in TypeScript type annotations, or HTML-like comments).

fix

For scenarios requiring absolute strict ECMAScript compliance, be aware of these limitations and consider a full-fledged AST parser. For most general-purpose tokenization (like syntax highlighting), `js-tokens`'s leniency is often an advantage.

affects: *

gotchaWhile designed to 'never fail', extremely large or malformed inputs (e.g., string literals with millions of repeated characters) can cause the underlying JavaScript regex engine to throw a `RangeError: Maximum call stack size exceeded` or similar error. This is a limitation of the regex engine, not `js-tokens`'s logic.

fix

Avoid processing extraordinarily large single tokens. For inputs with potential extreme length, consider pre-validation or chunking strategies if feasible.

affects: *

Errors

Common errors & fixes

SyntaxError: Cannot use import statement outside a module

Attempting to use ES module `import` syntax (`import jsTokens from 'js-tokens'`) in a CommonJS (`.js` file without `"type": "module"` in `package.json`) environment, especially on older Node.js versions.

fix

For CommonJS projects, either enable `"type": "module"` in your `package.json` (requires Node.js 12+), or use dynamic `await import('js-tokens')`. For older Node.js versions or strict CommonJS, you might need Node.js 20.19.0+ to directly `require('js-tokens')`. Otherwise, refactor your project to use ES Modules.

TypeError: jsTokens is not a function

This usually occurs when attempting to use `require('js-tokens')` in a CommonJS context for `js-tokens` versions that export a default ES module, and Node.js doesn't correctly resolve the default export.

fix

In CommonJS (for Node.js < 20.19.0 or without `"type": "module"`): use `const { default: jsTokens } = require("js-tokens");` or `const jsTokens = require("js-tokens").default;`. In ESM: ensure you use `import jsTokens from "js-tokens";`. For TypeScript, ensure `"esModuleInterop": true` in your `tsconfig.json`.

Upgrade

Version history

10.0.0latest on npm

Audit

Dependencies

No dependency data recorded yet.

Agent activity

4 hits · last 30 days

node

Resources

githubhttps://github.com/lydell/js-tokens ↗

packagehttps://www.npmjs.com/package/js-tokens ↗