ASTTokens is a Python library that annotates Abstract Syntax Trees (ASTs) with the positions of tokens and text in the source code that generated them. This functionality is essential for tools that perform source code transformations, such as automated refactoring or syntax highlighting. The current version is 3.0.1, released on November 15, 2025. The library is actively maintained with a stable release cadence, ensuring ongoing support and updates.
Install & Compatibility
Where this runs
tested against v3.0.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.925 runs
installs and imports cleanly · install 0.0s · import 0.018s · 17.9MB
glibcpy 3.10–3.925 runs
installs and imports cleanly · install 1.5s · import 0.016s · 18MB
16MB installed
● package 16MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
ASTTokens
✓ from asttokens import ASTTokens
Direct import of ASTTokens class for annotation tasks
ASTText
✓ from asttokens import ASTText
Import ASTText for text-based AST annotations
LineNumbers
✓ from asttokens import LineNumbers
Import LineNumbers for line and column number conversions
This example demonstrates how to parse and annotate source code using ASTTokens, and how to access the annotated AST and token positions.
import asttokens
import ast
# Sample source code
source = "Robot('blue').walk(steps=10*n)"
# Parse and annotate the source code
atok = asttokens.ASTTokens(source, parse=True)
# Access the annotated AST
tree = atok.tree
# Retrieve the first token of the first node
first_token = tree.body[0].first_token
# Print the start position of the first token
print(f"Start position: {first_token.startpos}")
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'asttokens'
The 'asttokens' module is not installed in the Python environment.
fixInstall the module using pip: 'pip install asttokens'.
ImportError: cannot import name 'formatargspec' from 'inspect'
The 'formatargspec' function was removed from the 'inspect' module in Python 3.11, causing compatibility issues with 'asttokens'.
fixUpdate 'asttokens' to a version compatible with Python 3.11 or later.
AttributeError: module 'asttokens' has no attribute 'ASTTokens'
The 'ASTTokens' class is not directly accessible from the 'asttokens' module due to incorrect import.
fixImport the class correctly: 'from asttokens import ASTTokens'.
ImportError: cannot import name 'ASTText' from 'asttokens'
This error typically occurs when using an older version of the `asttokens` library where the `ASTText` class was not directly exposed in the top-level package, while attempting to use code or examples written for a newer API version.
fixUpgrade `asttokens` to the latest version (e.g., `pip install --upgrade asttokens`) to ensure compatibility with recent API changes where `ASTText` is directly importable.
AttributeError: 'NoneType' object has no attribute 'body'
The `ASTTokens` object was initialized without setting the `parse=True` argument or providing an already parsed AST `tree`, causing its internal `tree` attribute to be `None`. Subsequent attempts to access attributes like `body` on the non-existent tree will fail.
fixInitialize `ASTTokens` with `parse=True` to automatically parse the source code, or explicitly pass an `ast.Module` object to the `tree` argument if you've parsed it separately. Example: `atok = asttokens.ASTTokens(source, parse=True)`.
Audit
Dependencies
astroidoptionalProvides additional AST capabilities for enhanced functionality