Registry / other / lindera-tokenizer

lindera-tokenizer

library0.32.3rscratesunverified

A morphological analysis library.

# Cargo.toml
[dependencies]
lindera-tokenizer = "0.32.3"

github ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · LINDERA-TOKENIZER

lindera-tokenizer

otherrustv0.32.3

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Tokenizer

✓ use lindera_tokenizer::tokenizer::Tokenizer;

Tokenizes Japanese text using Lindera.

use lindera_tokenizer::tokenizer::Tokenizer; use lindera_core::LinderaResult; fn main() -> LinderaResult<()> { let tokenizer = Tokenizer::new()?; let tokens = tokenizer.tokenize("すもももももももものうち")?; for token in tokens { println!("{}", token.text); } Ok(()) }

Debug

Known issues

gotchaRequires dictionary files; may need `lindera-dictionary` feature or external dictionary.

fix

Enable the `ipadic` feature or provide a custom dictionary path.

affects: >=0.32.0

Upgrade

Version history

0.32.3latest on crates.io

Audit

Dependencies

No dependency data recorded yet.

Agent activity

8 hits · last 30 days

node

Resources

githubhttps://github.com/lindera-morphology/lindera ↗

homepagehttps://github.com/lindera-morphology/lindera ↗

docshttps://docs.rs/lindera-tokenizer/ ↗