langid.py is a standalone Language Identification (LangID) tool for Python. It comes pre-trained on 97 languages and is designed for fast, domain-insensitive classification with minimal dependencies. It can be used as a simple Python library or deployed as a web service. The current version is 1.1.6, and while the original project maintains stability, a community-driven fork (`py3langid`) exists for modern Python 3 performance optimizations.
pip install langidVerified import paths — ran on the pinned version, not inferred.
The `langid.classify()` function takes a string and returns a tuple containing the predicted ISO 639-1 language code and an unnormalized log-probability estimate. For normalized probabilities (0-1), a `LanguageIdentifier` instance must be explicitly configured.
To obtain normalized probabilities (0-1), you must explicitly initialize a `LanguageIdentifier` instance with `norm_probs=True`. Example: `from langid.langid import LanguageIdentifier, MODEL_FILE; identifier = LanguageIdentifier.from_pickled_model(MODEL_FILE, norm_probs=True); identifier.classify(text)`.
Users needing to train custom models in modern Python environments should be aware that the original `langid` library's training tools are not Python 3 compatible. Consider using the `py3langid` fork, which has adapted training scripts (though noted as untested), or an alternative library for custom training.
Initialize the `langid` module once within your application (e.g., `import langid`) and then call `langid.classify()` multiple times. For batch processing of files, the library offers a batch mode (`-b` flag) that utilizes multiprocessing.
For new Python 3.6+ projects or performance-critical applications, consider using `py3langid`. It's designed as a drop-in replacement: `pip install py3langid` and then `import py3langid as langid`.
Ensure that all input to `langid.classify()` is a clean Unicode string. Always validate and potentially preprocess inputs, especially when sourcing text from databases or other external systems, to prevent unexpected formatting or encoding issues.
Install the package using pip: `pip install langid`
First, upgrade your pip installer: `python -m pip install --upgrade pip`. Then, retry the installation: `pip install langid`. If the problem persists for very old Python environments, you might try `pip install --pre langid` (though generally not necessary for version 1.1.6).
Ensure you have installed the package correctly using `pip install langid` in a clean Python 3 environment. If using the community-driven `py3langid` fork, the import statement is `import py3langid as langid`.
Install the Python development packages appropriate for your operating system. For Debian/Ubuntu, use `sudo apt-get install python3-dev` (or `python-dev` for Python 2). For CentOS/RHEL, use `sudo yum install python3-devel` (or `python-devel`).