Registry / ai-ml / bnunicodenormalizer

bnunicodenormalizer

library0.1.7pypypi✓ verified 85d ago

bnunicodenormalizer (v0.1.7) is a Python library designed for normalizing Bangla Unicode text. It provides tools to clean and standardize Bangla text by addressing inconsistent character representations, digit forms, and other common challenges, making the text suitable for various Natural Language Processing (NLP) tasks. The library saw active development in mid-2023 and is currently in maintenance.

ai-ml

pip install bnunicodenormalizer

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · BNUNICODENORMALIZE

bnunicodenormalizer

ai-mlpythonv0.1.7

Install

1.5s avg

Import

8ms

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.7 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 0.007s · 17.9MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 1.5s · import 0.003s · 18MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Normalizer

✓ from bnunicodenormalizer import Normalizer

Demonstrates how to initialize the Normalizer and use it to process a simple Bangla text string. The default initialization attempts to load necessary mapping files from the installed package directory.

from bnunicodenormalizer import Normalizer # Initialize the normalizer. # By default, it attempts to load 'romanize_map.json' from its package directory. bn_normalize = Normalizer() text_to_normalize = "এই টেস্টিং টেক্সট। ১০০ টাকা ।" result = bn_normalize(text_to_normalize) normalized_text = result["normalized_text"] print(f"Original: {text_to_normalize}") print(f"Normalized: {normalized_text}") # The result dictionary might also contain 'detected_lang' # if fasttext is enabled and detects it. # print(f"Detected language: {result.get('detected_lang', 'N/A')}")

Debug

Known issues

gotchaThe `Normalizer` class can optionally take a `romanize_mapping_path` argument. If a custom path is provided and is incorrect or the file is missing, it will result in a `FileNotFoundError`. If not provided, it attempts to load a default file from the package installation directory.

fix

Ensure the provided `romanize_mapping_path` is an absolute and correct path to a valid JSON file, or omit the argument to use the default package-provided mapping.

affects: All versions

breakingAs a `0.x.x` version library, minor version increments (e.g., from 0.1.x to 0.2.x) can introduce breaking changes without adhering strictly to SemVer, though no explicit breaking changes are documented between recent `0.1.x` versions.

fix

Pin your dependency to an exact version (`bnunicodenormalizer==0.1.7`) in `requirements.txt` or `pyproject.toml` to prevent unexpected updates. Review release notes for new versions before upgrading.

affects: All `0.x.x` versions

gotchaThe library has a direct dependency on `fasttext`. Installing `fasttext` can sometimes be challenging due to its native dependencies (e.g., C++ compiler). If `fasttext` fails to install correctly, the language detection features of `bnunicodenormalizer` will be unavailable or may cause errors, even if normalization functions still work.

fix

If `fasttext` installation fails, consult the `fasttext` documentation for system-specific prerequisites (e.g., `build-essential` on Debian/Ubuntu, XCode command line tools on macOS). You may need to install it separately first (`pip install fasttext`).

affects: All versions

Errors

Common errors & fixes

FileNotFoundError: [Errno 2] No such file or directory: '.../bnunicodenormalizer/romanize_map.json'

The `Normalizer` tried to load the default `romanize_map.json` but could not find it, often due to an incomplete installation or running from a non-standard environment.

fix

Ensure `bnunicodenormalizer` is properly installed via `pip install bnunicodenormalizer`. If providing a custom `romanize_mapping_path`, double-check the file existence and permissions for that path.

ModuleNotFoundError: No module named 'bnunicodenormalizer'

The `bnunicodenormalizer` package is not installed in the active Python environment.

fix

Install the package using `pip install bnunicodenormalizer`. If using virtual environments, ensure your IDE or terminal is activated to the correct environment.

AttributeError: 'dict' object has no attribute 'normalized_text'

The output of `bn_normalize(text)` is a dictionary. You are attempting to access a key as an attribute.

fix

Access dictionary keys using square bracket notation, e.g., `result['normalized_text']`. The returned dictionary structure is `{'normalized_text': '...', 'detected_lang': '...'}` (if language detection is enabled).

Upgrade

Version history

0.1.7latest on PyPI · released May 26, 2024

Audit

Dependencies

logururequiredLogging functionality

tqdmrequiredProgress bars for operations

regexrequiredAdvanced regular expression matching

fasttextrequiredOptional dependency for language detection functionality, can be heavy

Agent activity

14 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/mnansary/bnUnicodeNormalizer ↗

packagehttps://pypi.org/project/bnunicodenormalizer/ ↗

homepagehttps://bnunicode-normalizer.readthedocs.io ↗