Install & Compatibility
Where this runs
tested against v6.2.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.020s · 17.8MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 1.6s · import 0.016s · 18MB
16MB installed
● package 16MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
src_to_tokens
✓ from tokenize_rt import src_to_tokens
Function to convert source text to a list of tokens.
tokens_to_src
✓ from tokenize_rt import tokens_to_src
Function to convert a list of tokens back to source text.
Token
✓ from tokenize_rt import Token
Dataclass representing a token, including name, source, and offset.
ESCAPED_NL
✓ from tokenize_rt import ESCAPED_NL
Constant for the backslash-escaped newline token.
UNIMPORTANT_WS
✓ from tokenize_rt import UNIMPORTANT_WS
Constant for the unimportant whitespace token.
This quickstart demonstrates the core round-tripping functionality of `tokenize-rt`. It tokenizes a given Python source string using `src_to_tokens`, then converts the token stream back into source using `tokens_to_src`, verifying that the output matches the input.
from tokenize_rt import src_to_tokens, tokens_to_src
def roundtrip_code(code: str) -> str:
tokens = src_to_tokens(code)
# You can now inspect or modify 'tokens'
# For example, let's print them
for token in tokens:
print(f"Token(name={token.name!r}, src={token.src!r}, line={token.line}, offset={token.utf8_byte_offset})")
# Convert back to source
return tokens_to_src(tokens)
example_code = 'def foo(bar):
if bar: # a comment
return "hello world"
'
roundtripped_code = roundtrip_code(example_code)
print("\nOriginal Code:\n", example_code)
print("\nRoundtripped Code:\n", roundtripped_code)
assert example_code == roundtripped_code
Debug
Known issues
gotchatokenize-rt intentionally introduces additional token types (`ESCAPED_NL`, `UNIMPORTANT_WS`) and normalizes certain aspects (e.g., string prefixes, Python 2 literals in Python 3). This means its token stream will differ from that produced by the standard library's `tokenize` module, and direct comparisons or expectations of identical token streams should be adjusted.fixBe aware of the specific token types and normalizations added by `tokenize-rt` when porting code from or comparing with `stdlib.tokenize`.
affects: All versions
gotchaWhen reading Python source from a file, `src_to_tokens` (similar to `stdlib.tokenize.tokenize`) expects a callable `readline` function that returns lines as *bytes*, not a file object directly. If you open a file, ensure it's in binary read mode (`'rb'`) and pass `file_object.readline` to `src_to_tokens`.fixOpen files in binary mode: `with open('your_file.py', 'rb') as f: tokens = src_to_tokens(f.readline)`. affects: All versions
breakingThe underlying `stdlib.tokenize` module introduced a breaking change in tokenization of f-strings between Python 3.11 and Python 3.12 due to the formalization of PEP 701. While `tokenize-rt` aims to roundtrip reliably, tools built on fine-grained f-string token introspection might need careful review if moving between these Python versions.fixTest your application thoroughly across Python 3.11 and 3.12 if f-string tokenization details are critical. `tokenize-rt` generally handles this, but custom logic relying on specific token sequences might be affected.
affects: Python 3.11 to 3.12 runtime environments (and tokenize-rt versions used within them).
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'tokenize_rt'
The `tokenize-rt` library has not been installed in your Python environment or is not accessible in the current path.
fixInstall the library using pip: `pip install tokenize-rt`
AttributeError: module 'tokenize' has no attribute 'UNIMPORTANT_WS'
You are trying to access the `UNIMPORTANT_WS` token constant from the standard library's `tokenize` module, which does not define it. This constant is specific to `tokenize-rt`.
fixImport `UNIMPORTANT_WS` from the `tokenize_rt` library: `from tokenize_rt import UNIMPORTANT_WS`
AttributeError: module 'tokenize' has no attribute 'roundtrip_tokenize'
You have imported the standard library's `tokenize` module but are attempting to call `roundtrip_tokenize`, which is a function provided by the `tokenize_rt` library.
fixImport `tokenize_rt` and use its `roundtrip_tokenize` function: `import tokenize_rt; tokenize_rt.roundtrip_tokenize(...)`
AttributeError: module 'tokenize_rt' has no attribute 'tokenize'
You are trying to call a top-level `tokenize` function from the `tokenize_rt` module, but `tokenize_rt` uses `src_to_tokens` for tokenizing source code strings.
fixUse `tokenize_rt.src_to_tokens` instead of `tokenize_rt.tokenize`: `import tokenize_rt; tokenize_rt.src_to_tokens(source_code)`
Upgrade
Version history
6.2.0latest on PyPI · released May 23, 2025
Audit
Dependencies
pythonrequiredCore runtime dependency, requires Python >=3.9.