jieba

library0.42.1pypypi✓ verified 22d ago

Jieba is a popular open-source Python library for Chinese word segmentation, often referred to as 'jieba' (meaning 'to cut' or 'to slice'). It provides various segmentation modes, including dictionary-based, HMM (Hidden Markov Model), and a newer deep learning mode based on PaddlePaddle. The current version is 0.42.1, with an active development cadence, releasing several updates per year to address bugs and add features.

ai-ml data

pip install jieba

github ↗package ↗

INSTALL

IMPORT

SIG · JIEBA

jieba

ai-mlpythonv0.42.1

Install

5.5s avg

Import

4504ms

Disk

60MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.42.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 3.904s · 74.3MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 5.5s · import 5.103s · 75MB

60MB installed

● package 60MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

jieba

✓ import jieba

The primary module containing all core functions.

This quickstart demonstrates the basic usage of Jieba for Chinese word segmentation, showing the default, full, and search engine modes. It also includes a comment on how to load a custom user dictionary.

import jieba # Default segmentation mode (HMM + dictionary) text = "我来到北京清华大学" seg_list = jieba.cut(text, cut_all=False) print(f"Default Mode: {'/'.join(seg_list)}") # Full segmentation mode (all possible words) seg_list_all = jieba.cut(text, cut_all=True) print(f"Full Mode: {'/'.join(seg_list_all)}") # Search engine mode (short words for indexing) seg_list_search = jieba.cut_for_search(text) print(f"Search Engine Mode: {'/'.join(seg_list_search)}") # Custom dictionary loading # You would typically have a file named user.dict in the same directory # with words and their frequencies/parts of speech, e.g., # 结巴 3 # 人工智能 5 # jieba.load_userdict('user.dict')

Debug

Known issues

gotchaUsing the deep learning Paddle mode (`use_paddle=True`) requires `paddlepaddle` to be installed separately and can be significantly more resource-intensive and slower than the default HMM mode. Ensure `paddlepaddle` is installed (`pip install jieba[paddle]`) and monitor performance.

fix

Install `paddlepaddle` using `pip install jieba[paddle]`. For performance-critical applications, benchmark Paddle mode against default or HMM-only mode (`jieba.cut(text, HMM=False)`).

affects: >=0.40

gotchaPrior to v0.42, passing an empty string to `jieba.cut` in Paddle mode (`use_paddle=True`) could lead to a coredump. While fixed in v0.42, users on older versions or those not updating should ensure input strings are not empty when using this mode.

fix

Update Jieba to v0.42 or later. Always validate input strings to ensure they are non-empty before passing them to segmentation functions, especially with `use_paddle=True`.

affects: <0.42

gotchaThe `cut_all=True` (full mode) has historically had issues with correctly segmenting mixed English and Chinese text, and in some older versions (prior to v0.42), could potentially drop characters. While many issues have been addressed, users requiring absolute precision in mixed text might need to verify output or consider other modes.

fix

Update Jieba to the latest version (v0.42.1 or newer). For critical applications, manually inspect results for edge cases or consider using the default mode and post-processing.

affects: <0.42

gotchaWhen working with custom `Tokenizer` instances, ensure that `tokenizer.add_word()` and `tokenizer.del_word()` are used on the instance itself. Prior to v0.40, `add_word` could incorrectly affect the global default `Tokenizer`. Also, custom dictionaries with hyphens (`-`) were buggy before v0.40.

fix

Update Jieba to v0.40 or later. Always instantiate a `Tokenizer` and call its methods for custom dictionary management if you want isolated behavior. Ensure custom dictionary files are UTF-8 encoded.

affects: <0.40

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'jieba'

This error occurs when the 'jieba' library is not installed in the current Python environment or the environment running the script does not have it installed.

fix

Ensure you are in the correct Python environment and install 'jieba' using pip: `pip install jieba` or `pip3 install jieba`.

AttributeError: module 'jieba' has no attribute 'cut'

This typically happens when a Python script file is named 'jieba.py', causing Python to import the user's own file instead of the actual installed 'jieba' library.

fix

Rename your Python script file to something other than `jieba.py` (e.g., `my_segmenter.py`) to avoid a naming conflict.

Import error, cannot find paddle.fluid and jieba.lac_small.predict module. Now, back to jieba basic cut......

This error appears when `jieba.enable_paddle()` is called, but the PaddlePaddle deep learning framework is either not installed, not installed correctly, or its version is incompatible with the 'jieba' library's PaddlePaddle mode.

fix

Install `paddlepaddle-tiny` or the full `paddlepaddle` library, ensuring compatibility. For example: `pip install paddlepaddle-tiny==1.6.1` or `pip install paddlepaddle --upgrade`.

UnicodeDecodeError: 'gbk' codec can't decode byte 0xXX in position Y: illegal multibyte sequence

This encoding error occurs when attempting to read a text file (often containing Chinese characters) using an incorrect codec, frequently 'gbk' as the default on Windows, while the file is actually encoded in 'UTF-8' or another format.

fix

When opening the file, explicitly specify the correct encoding, typically 'utf-8', like so: `with open('your_file.txt', 'r', encoding='utf-8') as f:`.

UnicodeDecodeError: 'gbk' codec can't decode byte 0x... in position ...: illegal multibyte sequence

This error occurs when jieba or underlying Python file operations attempt to read text (e.g., from a user dictionary file or input string) using an incorrect encoding (often GBK on some Windows systems), but the content is actually UTF-8 or another encoding.

fix

Ensure all text files are saved with UTF-8 encoding and explicitly specify `encoding='utf-8'` when opening files or processing strings.

Upgrade

Version history

0.42.1latest on PyPI · released Jan 20, 2020

Audit

Dependencies

paddlepaddleoptionalRequired for `use_paddle=True` deep learning segmentation mode.

Agent activity

59 hits · last 30 days

node

Resources

githubhttps://github.com/fxsjy/jieba ↗

packagehttps://pypi.org/project/jieba/ ↗