wordninja

library2.0.0pypypi✓ verified 24d ago

Wordninja is a Python library that probabilistically splits concatenated words based on English Wikipedia uni-gram frequencies. It is designed to segment strings like 'imateapot' into ['im', 'a', 'teapot']. The current version is 2.0.0, released in August 2019, with a focus on stability rather than active new feature development.

data

pip install wordninja

github ↗package ↗

INSTALL

IMPORT

SIG · WORDNINJA

wordninja

datapythonv2.0.0

Install

2.5s avg

Import

313ms

Disk

17MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.0.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.334s · 19.7MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 2.5s · import 0.292s · 20MB

17MB installed

● package 17MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

split

✓ import wordninja wordninja.split('text')

✗ from wordninja import split

The primary function `split` is accessed directly from the imported `wordninja` module, not imported directly.

Demonstrates how to import the library and use the `split` function to segment a concatenated string into a list of words.

import wordninja split_words = wordninja.split('thisisateststring') print(split_words) split_phrase = wordninja.split('hellofromtheotherside') print(split_phrase)

Debug

Known issues

gotchaThe default `wordninja.split()` function removes punctuation from the input string and does not retain it in the output. For example, 'hello.world' becomes ['hello', 'world'].

fix

To preserve punctuation, you may need to pre-process the text, or consider using the `wordninja-enhanced` fork (e.g., `pip install wordninja-enhanced`) which includes punctuation preservation and other features not present in the original library.

affects: 2.0.0

gotchaThe probabilistic model can sometimes produce 'over-aggressive' or incorrect splits for specific words or abbreviations, such as 'patreon' splitting into ['pat', 're', 'on'] or 'esg' into ['e', 's', 'g'].

fix

For critical applications, consider reviewing results for specific edge cases or exploring custom language models. The library supports custom language models if provided as gzipped text files with one word per line in decreasing order of probability.

affects: 2.0.0

gotchaThe library is primarily designed for English. While custom language models are supported, creating them requires a specific gzipped text file format (one word per line, decreasing order of probability).

fix

If working with non-English languages or specific domains, ensure your custom word list adheres strictly to the required format. The `wordninja-enhanced` fork offers out-of-the-box support for several additional languages.

affects: 2.0.0

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'wordninja'

The wordninja library is not installed in the current Python environment or the interpreter cannot find it.

fix

Install the library using pip: `pip install wordninja`

IndexError: list index out of range

The wordninja library failed to load its internal language model (dictionary data file), often due to installation issues, corrupted files, or insufficient permissions.

fix

Reinstall wordninja using `pip install --force-reinstall wordninja` or ensure proper file permissions for the package installation directory.

TypeError: expected string or bytes-like object

The wordninja.split() function was called with an argument that is not a string, such as a number, list, or None.

fix

Ensure the argument passed to `wordninja.split()` is a string, for example: `wordninja.split('exampletext')`

ImportError: cannot import name 'wordninja' from 'wordninja'

The user is attempting a `from ... import ...` style import for the main module, which is not the standard way to import wordninja when using its top-level functions.

fix

Use `import wordninja` to import the library, and then call functions like `wordninja.split()`.

Upgrade

Version history

2.0.0latest on PyPI · released Aug 10, 2019

Audit

Dependencies

No dependency data recorded yet.

Agent activity

42 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/keredson/wordninja ↗

packagehttps://pypi.org/project/wordninja/ ↗