Registry /
serialization / json-stream-rs-tokenizer
Install & Compatibility
Where this runs
tested against v0.5.3 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.000s · 18.4MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 1.7s · import 0.000s · 20MB
17MB installed
● package 17MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
RustTokenizer
✓ from json_stream_rs_tokenizer import RustTokenizer
✗ from json_stream.tokenizer import RustTokenizer
The Rust tokenizer is exposed directly from `json_stream_rs_tokenizer`, not an internal `json_stream` submodule.
load
✓ from json_stream_rs_tokenizer import load
This provides a convenience wrapper around `json_stream.load` that pre-configures the Rust tokenizer.
This quickstart demonstrates how to explicitly use the `RustTokenizer` with `json-stream`'s `load` function and how to use the convenience `load` wrapper provided by this package.
from io import StringIO
from json_stream import load as json_stream_load
from json_stream_rs_tokenizer import RustTokenizer
# Example JSON data
json_buf = StringIO('{ "a": [1,2,3,4], "b": [5,6,7] }')
# Explicitly use the Rust tokenizer with json_stream.load
d = json_stream_load(json_buf, tokenizer=RustTokenizer)
print(f"Keys: {list(d.keys())}") # Output: Keys: ['a', 'b']
for k, l in d.items():
print(f"{k}: {' '.join(str(n) for n in l)}")
# Expected output for 'a': a: 1 2 3 4
# Alternatively, use the convenience wrapper from json_stream_rs_tokenizer
json_buf_alt = StringIO('{ "x": "hello", "y": "world" }')
from json_stream_rs_tokenizer import load as rs_load
data_alt = rs_load(json_buf_alt)
print(f"Value of x: {data_alt['x']}") # Output: Value of x: hello
Debug
Known issues
gotchaStarting with `json-stream` version 2.0, this tokenizer is used automatically if available. Explicitly installing `json-stream-rs-tokenizer` or passing `RustTokenizer` to `json_stream.load()` is usually unnecessary unless you are on an older `json-stream` version or need a specific configuration.fixEnsure `json-stream` is updated. If you are using `json-stream` 2.0+, simply installing `json-stream-rs-tokenizer` is sufficient for it to be used implicitly.
affects: json-stream >= 2.0
breakingInstallation from source requires a Rust toolchain. If a prebuilt wheel is not available for your platform and Python version, the installation process will attempt to build from source. A failed Rust build might still report a successful Python package installation, but `RustTokenizer` will not be importable.fixEnsure a Rust toolchain is installed and accessible in your environment (`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`). Increase pip verbosity (`pip install -vv json-stream-rs-tokenizer`) to see detailed build logs and diagnose Rust compilation errors.
affects: All versions (when installing from source)
gotchaWhen installed in editable/development mode, the Rust library might be compiled in debug mode, which can make it *slower* than the pure-Python tokenizer.fixTo ensure release mode compilation, verify installation commands use `--release`. Check `pip install -v` output for Rust compilation flags.
affects: All versions (when installing in debug/development mode)
gotchaFor PyPy, the performance improvement from `json-stream-rs-tokenizer` is significantly lower (1.0-1.5x) compared to CPython (4-10x).fixBe aware of this limitation if deploying on PyPy; the benefits might not be as pronounced.
affects: All versions (on PyPy)
gotchaWhen parsing mixed data with `json-stream.load(..., correct_cursor=True)`, keeping track of the exact stream position for un-seekable streams incurs a significant performance cost.fixAvoid `correct_cursor=True` for un-seekable streams unless absolutely necessary for mixed data parsing. Consider refactoring data formats or using seekable streams if possible.
affects: All versions (when using `correct_cursor=True` with un-seekable streams)
Errors
Common errors & fixes
ERROR: Failed building wheel for json-stream-rs-tokenizer
pip attempted to build `json-stream-rs-tokenizer` from its source distribution, but the Rust toolchain (compiler and Cargo) was not installed or correctly configured on the system, preventing the native extension from compiling.
fixInstall the Rust toolchain by following the instructions on the official Rust website (rustup.rs), then retry the installation command (e.g., `pip install json-stream`).
ModuleNotFoundError: No module named 'json_stream_rs_tokenizer.RustTokenizer'
This error occurs when the `json-stream-rs-tokenizer` package was installed, but its underlying Rust component failed to compile during installation, or a pre-built wheel was not available for your specific platform. As a result, the `RustTokenizer` class cannot be imported. The `json-stream` library will fall back to its pure-Python tokenizer if this happens.
fixEnsure the Rust toolchain is installed on your system (e.g., via `rustup init`) and then reinstall `json-stream-rs-tokenizer` explicitly, or reinstall `json-stream` to ensure it attempts to pick up the Rust tokenizer again. You can use `pip install json-stream-rs-tokenizer --force-reinstall -vv` to see detailed build errors. Ensure your Python environment and operating system are supported for pre-built wheels if you don't wish to compile from source.
ERROR: Could not build wheels for json-stream-rs-tokenizer, which is required to install pyproject.toml-based projects
This is a general `pip` error indicating that the build process for `json-stream-rs-tokenizer` (which uses a `pyproject.toml` for its build system) failed. The most common reason for this failure is the absence of a Rust toolchain required to compile its native Rust extension.
fixInstall the Rust toolchain (Rust compiler and Cargo) by following the instructions on the official Rust website (rustup.rs), then retry the package installation.
Upgrade
Version history
0.5.3latest on PyPI · released Aug 10, 2026
Audit
Dependencies
json-streamrequiredThis library provides a faster tokenizer specifically for `json-stream`. `json-stream` v2.0+ automatically uses it if installed.