Registry / ai-ml / spacy-alignments

spacy-alignments

JSON →
library0.9.2pypypi✓ verified 84d ago

spacy-alignments is a Python library that provides efficient tokenization alignment capabilities, particularly useful for integrating different NLP tools like spaCy and transformer models. It offers Python bindings for Yohei Tamura's highly performant Rust `tokenizations` library. The current version is 0.9.2, with releases primarily focused on supporting new Python versions and underlying PyO3 updates.

pip install spacy-alignments
INSTALL
IMPORT
SIG · SPACY-ALIGNMENTS
S
spacy-alignments
ai-mlpythonv0.9.2
Install
1.6s avg
Import
Disk
17MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v0.9.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.920 runs
installs and imports cleanly · install 0.0s · import 0.000s · 18.7MB
glibc
py 3.103.920 runs
installs and imports cleanly · install 1.6s · import 0.000s · 19MB
17MB installed
● package 17MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

get_alignments
from spacy_alignments import get_alignments
import spacy_alignments as tokenizations
get_charmap
from spacy_alignments import get_charmap
get_original_spans
from spacy_alignments import get_original_spans

The `get_alignments` function is the core of the library, providing bidirectional mapping between two sequences of tokens that may have undergone different tokenization or normalization processes.

import spacy_alignments as tokenizations # Example from spacy-alignments README/PyPI tokens_a = ["å", "BC"] tokens_b = ["abc"] # the accent is dropped (å -> a) and the letters are lowercased(BC -> bc) # Get alignment mappings for two different tokenizations a2b, b2a = tokenizations.get_alignments(tokens_a, tokens_b) print(f"Alignment from tokens_a to tokens_b: {a2b}") print(f"Alignment from tokens_b to tokens_a: {b2a}")
Debug
Known issues
breakingVersion 0.9.0 dropped support for Python 3.6.
fix
Upgrade to Python 3.9 or newer. The current version (0.9.2) supports Python >=3.9, <3.14.
affects: >=0.9.0
gotchaInstallation may require Rust compiler if pre-built binary wheels are not available for your platform and Python version.
fix
If `pip install spacy-alignments` fails with build errors, install Rust by following instructions on `rustup.rs` and ensure `cargo` is in your system's PATH.
affects: All versions
gotchaWhen using `spacy-transformers` v1.2 or newer, the alignment between spaCy tokens and transformer tokens for 'fast tokenizers' may differ from previous versions. This is because `spacy-transformers` now uses exact alignment from the underlying tokenizers instead of `spacy-alignments`'s heuristic method.
fix
Be aware of potential subtle changes in token alignments when upgrading `spacy-transformers`. Verify your downstream tasks if they rely on specific alignment behavior. This primarily impacts transformer pipelines within spaCy rather than direct `spacy-alignments` usage.
affects: spacy-transformers >=1.2
Upgrade
Version history
0.9.2latest on PyPI · released Jun 3, 2025
Audit
Dependencies
spacyoptionalDesigned to align tokenizations for spaCy and transformers, implicitly requiring spaCy for most use cases.
rustoptionalRequired to build the package from source if no pre-compiled binary wheels are available for your platform and Python version.
Agent activity
15 hits · last 30 days
node
10
OpenAI (training)
1
Resources
spacy-alignments — pip install spacy-alignments · libregistry