konoha

library5.7.0pypypi✓ verified 85d ago

Konoha is a Python library (v5.7.0) that provides a unified, easy-to-use interface for various Japanese tokenizers, including MeCab, Sudachi, and Sentencepiece. It allows developers to seamlessly switch between different tokenizers and also offers rule-based tokenizers (whitespace, character) and a sentence splitter. The library is actively maintained with its latest release in March 2026.

llm-agents ai-ml

pip install 'konoha[all]'

package ↗

INSTALL

IMPORT

SIG · KONOHA

konoha

llm-agentspythonv5.7.0

Install

10.9s avg

Import

590ms

Disk

631MB

Pass rate

4/ 10

Env Coverage4 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v5.7.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

8/12 runs

✓ 12.23s

py 3.11

8/12 runs

✓ 10.42s

py 3.12

8/12 runs

✓ 10.38s

py 3.13

8/12 runs

✓ 10.49s

py 3.9

8/12 runs

631MB installed

● package 631MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

WordTokenizer

✓ from konoha import WordTokenizer

SentenceTokenizer

✓ from konoha import SentenceTokenizer

Demonstrates basic word-level tokenization of a Japanese sentence using the `WordTokenizer` with MeCab. Ensure the necessary tokenizer is installed as an extra.

from konoha import WordTokenizer sentence = '自然言語処理を勉強しています' # Initialize with a supported tokenizer (e.g., MeCab) # Ensure 'konoha[mecab]' or 'konoha[all]' is installed tokenizer = WordTokenizer('MeCab') tokens = tokenizer.tokenize(sentence) print([token.surface for token in tokens])

Debug

Known issues

gotchaInstalling `konoha` without specifying extras (e.g., `pip install konoha`) will only install the sentence splitter, not any word tokenizers. To use tokenizers like MeCab or Sudachi, you must install `konoha` with the corresponding extra (e.g., `konoha[mecab]`) or `konoha[all]` for all supported tokenizers.

fix

Use `pip install 'konoha[all]'` or `pip install 'konoha[<tokenizer_name>]'` to include desired tokenizer dependencies.

affects: All versions

breakingThe API endpoint paths for the Docker quickstart (e.g., `/api/v1/tokenize`) changed in v4.6.4. Older `curl` commands or Docker configurations might fail.

fix

Refer to the latest documentation or GitHub README for updated API paths when using the Konoha Docker image. For example, check release notes at `https://github.com/himkt/konoha/releases/tag/v4.6.4`.

affects: >=4.6.4

gotchaWhen using the `Sentencepiece` tokenizer, you must provide a valid `model_path` argument to `WordTokenizer`. Omitting it will result in an error or unexpected behavior.

fix

Always pass the `model_path` argument when initializing `WordTokenizer('Sentencepiece', model_path="your/model.spm")`.

affects: All versions

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'MeCab'

The MeCab tokenizer dependency (mecab-python3) was not installed along with konoha.

fix

Install konoha with the mecab extra: `pip install 'konoha[mecab]'` or `pip install 'konoha[all]'`.

RuntimeError: 'sudachipy' is not installed. Please install it with 'pip install sudachipy'

The Sudachi tokenizer dependency (sudachipy and sudachidict_core) was not installed.

fix

Install konoha with the sudachi extra: `pip install 'konoha[sudachi]'` or `pip install 'konoha[all]'`.

TypeError: __init__() missing 1 required positional argument: 'model_path'

Attempted to initialize `WordTokenizer('Sentencepiece')` without providing the `model_path` argument.

fix

Provide the path to your Sentencepiece model file: `WordTokenizer('Sentencepiece', model_path="path/to/your/model.spm")`.

Upgrade

Version history

5.7.0latest on PyPI · released Mar 1, 2026

Audit

Dependencies

mecab-python3optionalRequired for MeCab tokenizer functionality.

sudachipyoptionalRequired for Sudachi tokenizer functionality.

sudachidict_coreoptionalRequired for Sudachi tokenizer functionality.

janomeoptionalRequired for Janome tokenizer functionality.

sentencepieceoptionalRequired for Sentencepiece tokenizer functionality.

nagisaoptionalRequired for Nagisa tokenizer functionality.

allennlpoptionalRequired for AllenNLP integration via 'all_with_integrations' extra.

Agent activity

11 hits · last 30 days

node

OpenAI (training)

Resources

packagehttps://pypi.org/project/konoha/ ↗