Install & Compatibility
Where this runs
tested against v0.16.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.13
✕ build_error
✕ build_error
92MB installed
● package 92MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
Compose
✓ from hojichar import Compose
✗ from hojichar.core import Compose
Compose is a top-level export since v0.10.0
AsyncCompose
✓ from hojichar import AsyncCompose
Introduced in v0.14.0
JSONLoader
✓ from hojichar.document_filters import JSONLoader
Part of document_filters module
JSONDumper
✓ from hojichar.document_filters import JSONDumper
✗ from hojichar.filters import JSONDumper
JSONDumper moved to hojichar.document_filters in v0.10.0
Build a text preprocessing pipeline that reads JSONL, filters Japanese text, discards documents with too many or too few letters, and outputs JSONL with metadata.
from hojichar import Compose, document_filters
from hojichar.filters import AcceptJapaneseFilter, DiscardTooManyFilter, DiscardTooFewLettersFilter
pipeline = Compose([
document_filters.JSONLoader(),
AcceptJapaneseFilter(),
DiscardTooManyFilter(max_filtered_num=10000),
DiscardTooFewLettersFilter(min_letters=10),
document_filters.JSONDumper(export_extras=True),
])
with open('input.jsonl', 'r') as f:
results = pipeline(f.read())
Debug
Known issues
breakingIn v0.15.0, the deduplication module was overhauled. GenerateDedupLSH now uses the Rust-based `rensa` engine. The old `MinHash` and `LSH` classes were removed. If you relied on the previous Python-only implementation, you must update imports and usage.fixUse `from hojichar.filters.deduplication import GenerateDedupLSH` and ensure `rensa` is installed (pip install rensa). For in-memory dedup without Rust, use `InlineDeduplicator`.
affects: >=0.15.0
deprecatedStatistics properties on filters (e.g., `.stats`) are deprecated since v0.15.3. Use the new `stats` module or access via pipeline-level statistics.fixMigrate to using `pipeline.stats` or individual filter stats via `filter.get_stats()`.
affects: >=0.15.3,<0.17.0
gotchaThe `JSONDumper` by default excludes extras key `'__init_stats'` from output. If you need that metadata, set `export_extras=True` and explicitly manage the extras dict.fixUse `JSONDumper(export_extras=True)` and ensure the extras dict does not contain `'__init_stats'` if you want it exported, or override the dumper.
affects: >=0.14.1
gotchaJapanese-language filters like `DiscardTooManyNouns` and `WordRepetitionRatioFilter` may cause segfaults on very large texts if `max_parse_chars` is not set. Defaults were adjusted in v0.15.1, but if you encounter segmentation faults, explicitly set a limit.fixSet `max_parse_chars` parameter (e.g., `DiscardTooManyNouns(max_parse_chars=1000000)`).
affects: >=0.15.0
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'hojichar.core'
Trying to import from old internal module structure that was restructured in v0.10.0.
fixUse the top-level public API: `from hojichar import Compose` instead of `from hojichar.core import Compose`.
AttributeError: module 'hojichar' has no attribute 'AsyncCompose'
Using a version older than v0.14.0 where AsyncCompose was introduced.
fixUpgrade to hojichar >= 0.14.0: `pip install 'hojichar>=0.14.0'`.
ImportError: cannot import name 'JSONDumper' from 'hojichar.filters'
JSONDumper was moved to `hojichar.document_filters` in v0.10.0.
fixUse `from hojichar.document_filters import JSONDumper`.
segfault (Fatal Python error: Segmentation fault) when using DiscardTooManyNouns on large text
fugashi parsing a very long text without parse length limit can crash.
fixSet `max_parse_chars` parameter, e.g., `DiscardTooManyNouns(max_parse_chars=500000)`.
Upgrade
Version history
0.16.0latest on PyPI · released Nov 13, 2025
Audit
Dependencies
fugashi[tagger]optionalRequired for Japanese-language filters like DiscardTooManyNouns and WordRepetitionRatioFilter
rensaoptionalRust-based engine used by GenerateDedupLSH for near-duplicate detection (v0.15.0+)