sql-tokenizer

library1.0.5jsnpmunverified

A lightweight SQL tokenizer that converts SQL statements into a list of tokens. Current stable version is 1.0.5, released in 2023. It is a simple, dependency-free package for basic SQL tokenization, handling whitespace, quotes, and indentation preservation. Ships TypeScript definitions. Useful for SQL parsing or syntax highlighting. Updates are infrequent. Unlike full SQL parsers, it only tokenizes without building an AST.

database

npm install sql-tokenizer

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · SQL-TOKENIZER

sql-tokenizer

databasejavascriptv1.0.5

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

sqlTokenizer

✓ import { sqlTokenizer } from 'sql-tokenizer'

✗ const sqlTokenizer = require('sql-tokenizer')

ESM-only package; CJS require will fail. Use dynamic import if needed.

sqlSpecialChars

✓ import { sqlSpecialChars } from 'sql-tokenizer'

✗ const sqlSpecialChars = require('sql-tokenizer').sqlSpecialChars

Must be imported as named export.

explicit type import

✓ import type { SqlTokenizer } from 'sql-tokenizer'

✗ import { SqlTokenizer } from 'sql-tokenizer'

Use type import for TypeScript type if needed.

Demonstrates basic usage: import, create tokenizer function, and tokenize a simple SQL statement.

import { sqlTokenizer } from 'sql-tokenizer' const tokenize = sqlTokenizer() const tokens = tokenize("SELECT * FROM users WHERE id = 1") console.log(tokens) // ['SELECT', ' ', '*', ' ', 'FROM', ' ', 'users', ' ', 'WHERE', ' ', 'id', ' ', '=', ' ', '1']

Debug

Known issues

gotchaPackage is ESM-only; cannot use require() in CommonJS projects.

fix

Use dynamic import or type: "module" in package.json.

affects: >=1.0.0

gotchaTokenizer does not perform syntax validation; malformed SQL will still produce tokens.

fix

Validate SQL separately if needed.

affects: >=1.0.0

gotchaWhitespace tokens are preserved as separate tokens (including spaces and newlines).

fix

Filter whitespace tokens if only meaningful tokens are needed.

affects: >=1.0.0

Errors

Common errors & fixes

Error [ERR_REQUIRE_ESM]: require() of ES Module ... not supported.

Using require() to import an ESM-only package.

fix

Use import statement or dynamic import() instead.

TypeError: sqlTokenizer is not a function

Incorrect import: default import instead of named import.

fix

Use `import { sqlTokenizer } from 'sql-tokenizer'`

Upgrade

Version history

1.0.5latest on npm

Audit

Dependencies

No dependency data recorded yet.

Agent activity

15 hits · last 30 days

node