bpemb

library0.3.6pypypi✓ verified 85d ago

BPEmb provides byte-pair encodings (BPE) from raw text and maps subword units to pre-trained embeddings for 275 languages. It's designed for NLP tasks requiring efficient subword tokenization and embedding. The current version is 0.3.6, with releases occurring sporadically based on updates to models or features.

ai-ml

pip install bpemb

package ↗homepage ↗

INSTALL

IMPORT

SIG · BPEMB

bpemb

ai-mlpythonv0.3.6

Install

9.5s avg

Import

281ms

Disk

291MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.3.6 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

build_error

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 9.5s · import 0.281s · 286MB

291MB installed

● package 291MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

BPEmb

✓ from bpemb import BPEmb

Demonstrates how to initialize BPEmb, encode a sentence into subwords, and retrieve embeddings for individual words or lists of words. The first initialization for a specific (lang, dim, vs) combination will trigger a large model download.

from bpemb import BPEmb # Initialize BPEmb for English, 100-dim embeddings, 100,000 vocabulary size # This will download the model the first time it's run. bpemb_en = BPEmb(lang="en", dim=100, vs=100000) # Encode a sentence into subword units encoded_sentence = bpemb_en.encode("This is a test sentence for bpemb.") print(f"Encoded sentence: {encoded_sentence}") # Get embeddings for a single word embedding = bpemb_en.embed("test") print(f"Embedding shape for 'test': {embedding.shape}") # Get embeddings for a list of words embeddings_list = bpemb_en.embed_words(["this", "is", "bpemb"]) print(f"Embeddings shape for word list: {embeddings_list.shape}")

Debug

Known issues

gotchaThe BPEmb constructor triggers large model downloads (hundreds of MBs to GBs) for each unique (language, dimension, vocabulary_size) combination upon first use. This can consume significant disk space and bandwidth.

fix

Ensure stable internet connection and sufficient disk space. Models are cached locally for subsequent uses. Consider using smaller `dim` or `vs` values if disk space is a concern.

affects: All versions

gotchaLoaded models can consume significant RAM (hundreds of MBs or more) depending on the chosen `dim` and `vs` parameters, potentially leading to `MemoryError` on systems with limited resources.

fix

If encountering `MemoryError` or performance issues, try reducing the `dim` (embedding dimension) and/or `vs` (vocabulary size) parameters during `BPEmb` initialization. Process large texts in batches if possible.

affects: All versions

gotchaOut-of-Vocabulary (OOV) words are handled differently by `encode` (which performs subword segmentation) and `embed` (which returns a zero vector for unknown words). Users might expect a unified behavior.

fix

Be aware of the distinct behaviors: `encode` will break unknown words into subwords using BPE, while `embed` will yield a zero vector for any word not in its vocabulary. Handle OOV words explicitly in your application logic based on the desired outcome.

affects: All versions

Errors

Common errors & fixes

urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

Network error preventing the download of pre-trained models. This can be due to no internet connection, firewall issues, or incorrect proxy settings.

fix

Verify your internet connection. Check firewall settings to ensure Python can make outbound connections. If behind a proxy, configure proxy settings for your environment or Python requests.

FileNotFoundError: [Errno 2] No such file or directory: '/home/user/.cache/bpemb/en_100000_100.model'

The cached model files for BPEmb were deleted, moved, or the cache directory is inaccessible/corrupted, and the library cannot locate them.

fix

Ensure the `~/.cache/bpemb` directory (or your custom cache path) is intact and accessible. If files are missing, `BPEmb` will attempt to re-download them automatically upon initialization.

ValueError: Language 'xx' not supported. Available languages are: ['en', 'de', ...]

An unsupported or incorrect language code was provided to the `BPEmb` constructor.

fix

Consult the `bpemb` documentation or its source code for an accurate list of supported language codes. Ensure the language code matches one of the officially available options (e.g., 'en' for English, 'es' for Spanish).

MemoryError

Attempting to load a very large model (high `dim` and `vs` parameters) or process an extremely large amount of text at once, exceeding available system RAM.

fix

Reduce the `dim` (embedding dimension) and/or `vs` (vocabulary size) parameters when initializing `BPEmb`. Consider processing text in smaller, manageable batches if possible to distribute memory load.

Upgrade

Version history

0.3.6latest on PyPI · released Oct 1, 2024

Audit

Dependencies

numpyrequiredNumerical operations and array handling for embeddings.

tqdmrequiredProgress bars for downloads and processing.

Agent activity

20 hits · last 30 days

node

OpenAI (training)

Resources

packagehttps://pypi.org/project/bpemb/ ↗

homepagehttps://nlp.h-its.org/bpemb ↗