pytokens

library0.4.1pypypi✓ verified 27d ago

Pytokens is an open-source Python library providing a fast, spec-compliant tokenizer for Python 3.14+ that is also capable of running on older Python versions (>=3.8). Currently at version 0.4.1, it appears to be actively maintained with recent releases focusing on packaging and development improvements.

serialization

pip install pytokens

github ↗package ↗

INSTALL

IMPORT

SIG · PYTOKENS

pytokens

serializationpythonv0.4.1

Install

1.7s avg

Import

43ms

Disk

17MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.4.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.044s · 18.5MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.7s · import 0.042s · 19MB

17MB installed

● package 17MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

pytokens

✓ import pytokens

Pytokens is primarily designed for command-line usage to tokenize Python files. This example demonstrates tokenizing a simple Python file. The output will be a stream of tokens (e.g., NAME, STRING, OP, etc.) representing the input file's content.

echo "print('Hello, World!')" > example.py python -m pytokens example.py

Debug

Known issues

gotchaPytokens is compiled with mypyc by default for performance. This means the installed module might be a compiled extension (.so or .pyd) rather than pure Python code, which can affect debugging or introspection.

fix

To disable mypyc compilation (e.g., for local development or debugging), set the environment variable PYTOKENS_USE_MYPYC=0 before installation (e.g., PYTOKENS_USE_MYPYC=0 pip install pytokens).

affects: >=0.4.0

gotchaWhile pytokens aims for Python 3.14+ spec compliance and runs on older Python versions (>=3.8), be aware that its tokenization behavior adheres to the 3.14+ specification. This might introduce subtle differences compared to the native tokenizer behavior of older Python interpreters.

fix

Ensure your environment or tests account for the 3.14+ tokenizer specification if running on older Python versions, especially if precise tokenizer behavior is critical for your application.

affects: All versions when running on Python < 3.14

breakingThe script contains shell commands (e.g., `echo`, `>`) which result in a `SyntaxError` when executed by the Python interpreter. This typically occurs when a shell command is mistakenly placed in a Python file or when a shell script is run with the Python interpreter.

fix

Ensure that Python scripts only contain valid Python syntax. If shell commands are necessary, they should be executed using Python's `subprocess` module or placed in a separate shell script and executed appropriately.

affects: All Python versions (as this is a fundamental language syntax error)

breakingThe script attempted to execute shell commands directly as Python code, leading to a `SyntaxError`. Python interpreters do not understand shell commands like `echo` or shell redirection (`>`).

fix

Ensure that shell commands are executed in a shell environment (e.g., by structuring the build/test script to correctly separate shell commands from Python script execution, or by using Python's `subprocess` module with `shell=True` if executing shell commands from Python code). If the file was intended to be a shell script, ensure it is executed by a shell interpreter.

affects: All versions

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'pytokens'

The `pytokens` library has not been installed in the current Python environment or the environment is not active.

fix

Install the library using pip: `pip install pytokens`

TypeError: an integer is required (got type dict)

The `pytokens.tokenize` function received an input that was not a string, which is required for the Python source code.

fix

Ensure the input passed to `pytokens.tokenize()` is always a string containing the Python source code. Example: `pytokens.tokenize("def func():\n    pass")`

ERROR: No matching distribution found for pytokens

Pip could not find a compatible pre-built package for the current Python version (must be >= 3.8) or platform.

fix

Ensure you are using Python 3.8 or newer; if the issue persists, check PyPI for available wheels for your specific environment.

ModuleNotFoundError: No module named 'pytokens.tokenize'

`tokenize` is a function directly available under the `pytokens` package, not a submodule, so it cannot be imported using dot notation as a nested module.

fix

Import the function directly from the package: `from pytokens import tokenize` or use `pytokens.tokenize` after `import pytokens`.

Upgrade

Version history

0.4.1latest on PyPI · released Jan 30, 2026

Audit

Dependencies

mypyoptionalUsed for type checking during development. Not a runtime dependency.

Agent activity

12 hits · last 30 days

node

Amazon

Resources

githubhttps://github.com/tusharsadhwani/pytokens ↗

packagehttps://pypi.org/project/pytokens/ ↗