BlingFire is a Python wrapper for a lightning-fast Finite State Machine (FSM) based Natural Language Processing (NLP) library developed by Microsoft. It is designed for high-performance text tokenization, multi-word expression matching, stemming, and lemmatization. Known for its speed, it often outperforms other NLP libraries like Hugging Face and SpaCy in tokenization tasks. The library supports various tokenization algorithms including pattern-based, WordPiece, Unigram LM, and BPE. The current version is 0.1.8, and it maintains an active release cadence with periodic updates adding new features and models.
pip install blingfireVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates basic sentence splitting and word tokenization using BlingFire's default models, which do not require explicit model loading. The output is a single string that can be split by newline for sentences or space for words.
Review code that processes token offsets, especially the first token's offset, when using models that might employ a dummy prefix.
Ensure required `.bin` model files are available and load them explicitly using `handle = load_model('./path/to/your_model.bin')` before calling functions like `tokenize_with_model` or `IdsToText`.Verify that the model being used with `IdsToText` explicitly supports ID-to-word conversion. Refer to the model's documentation or origin for compatibility.
For East Asian or other non-space-delimited languages, explicitly load and use a model specifically trained for that language or script. Consult the BlingFire documentation for available multilingual models.
No dependency data recorded yet.