py3langid

library0.3.0pypypi✓ verified 86d ago

py3langid is an actively maintained fork of the original `langid.py` library, specializing in fast and accurate language identification. It is optimized for Python 3 environments, featuring a modernized codebase and improved execution speeds. The library's current version is 0.3.0, with a release cadence that reflects ongoing enhancements and bug fixes.

ai-ml

pip install py3langid

github ↗package ↗

INSTALL

IMPORT

SIG · PY3LANGID

py3langid

ai-mlpythonv0.3.0

Install

3.6s avg

Import

294ms

Disk

90MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.3.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 0.290s · 90.1MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 3.6s · import 0.297s · 86MB

90MB installed

● package 90MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

classify

✓ from py3langid import classify

✗ import py3langid as langid

rank

✓ from py3langid import rank

set_languages

✓ from py3langid import set_languages

This quickstart demonstrates basic language classification using `py3langid.classify()`. It also shows how to use `LanguageIdentifier` for more control, such as enabling probability normalization to get scores between 0 and 1.

import py3langid as langid text_en = 'This text is in English.' lang, prob = langid.classify(text_en) print(f"Text: '{text_en}' -> Language: {lang}, Probability: {prob}") text_de = 'Dieser Text ist auf Deutsch.' lang, prob = langid.classify(text_de) print(f"Text: '{text_de}' -> Language: {lang}, Probability: {prob}") # Example with probability normalization from py3langid.langid import LanguageIdentifier, MODEL_FILE identifier = LanguageIdentifier.from_pickled_model(MODEL_FILE, norm_probs=True) text_norm = 'This should be enough text.' lang_norm, prob_norm = identifier.classify(text_norm) print(f"Text (normalized): '{text_norm}' -> Language: {lang_norm}, Normalized Probability: {prob_norm}")

langid --version

Debug

Known issues

breakingSupport for Python 3.6 and 3.7 was dropped in py3langid v0.3.0. Users on these older Python versions will need to upgrade their Python interpreter or stick to py3langid v0.2.x.

fix

Upgrade Python to 3.8 or newer, or pin `py3langid<0.3.0` in your project dependencies.

affects: >=0.3.0

breakingThe default Numpy data type for feature vectors changed from `uint32` to `uint16` in v0.2.0 for performance optimization. While generally transparent, this could affect applications sensitive to exact data types or those comparing results with older versions.

fix

If inconsistent results are observed or specific `uint32` typing is required, pass `datatype='uint32'` to the `classify()` method or `LanguageIdentifier` constructor (e.g., `langid.classify(text, datatype='uint32')`).

affects: >=0.2.0

gotchaThe original `langid.py` (and by extension `py3langid`) training scripts remain Python 2-only. Users expecting to retrain models with custom data using the provided tools might encounter compatibility issues with Python 3.

fix

For training new models, refer to the original `langid.py` project's documentation and Python 2 environment setup. `py3langid` primarily focuses on the classification aspect in Python 3.

affects: All versions

Upgrade

Version history

0.3.0latest on PyPI · released Jun 18, 2024

Audit

Dependencies

numpyrequiredRequired for numerical operations and efficient feature vector handling. Specific versions are required based on Python version.

Agent activity

11 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/adbar/py3langid ↗

packagehttps://pypi.org/project/py3langid/ ↗