justext

library3.0.2pypypi✓ verified 24d ago

justext is a heuristic-based boilerplate removal tool for HTML documents. It extracts the main content from web pages, discarding navigation, advertisements, and other extraneous elements. The current version is 3.0.2, and it typically releases updates for bug fixes and compatibility issues.

data

pip install justext

github ↗package ↗

INSTALL

IMPORT

SIG · JUSTEXT

justext

datapythonv3.0.2

Install

2.3s avg

Import

253ms

Disk

30MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v3.0.2 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.154s · 32.4MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 2.3s · import 0.150s · 33MB

30MB installed

● package 30MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

justext

✓ import justext

✗ from justext import justext # While technically possible, justext is often used as a module directly, and justext.justext() is the main function.

The primary function is justext.justext(). Accessing it via 'import justext' and then 'justext.justext()' is the most common pattern.

This quickstart demonstrates how to fetch an HTML document using `requests`, then pass its raw byte content to `justext.justext()` along with a predefined stoplist (e.g., 'English') to extract and print human-readable text, filtering out boilerplate.

import requests import justext # Example URL (replace with a real URL for actual testing) url = "https://www.python.org" try: response = requests.get(url, timeout=5) # justext expects bytes as input html_content = response.content # Get the English stoplist stoplist = justext.get_stoplist("English") # Process the HTML content paragraphs = justext.justext(html_content, stoplist) print(f"Extracted text from {url}:") for paragraph in paragraphs: if not paragraph.is_boilerplate: print(paragraph.text) except requests.exceptions.RequestException as e: print(f"Error fetching URL: {e}") except Exception as e: print(f"An error occurred: {e}")

justext --version

Debug

Known issues

breakingjustext v3.0.0 dropped support for Python 3.4 and older versions (including Python 2.x). Attempts to install or run on these versions will fail or lead to unexpected behavior.

fix

Upgrade to Python 3.5 or newer. For projects requiring older Python versions, consider pinning justext to < 3.0.0 (e.g., `justext<3.0.0`).

affects: < 3.0.0 (running on Python 3.4- or 2.x)

gotchaThe `justext.justext()` function expects raw HTML content as bytes (e.g., from `response.content`). Passing a decoded string (e.g., `response.text`) can lead to parsing errors or incorrect results due to encoding issues with `lxml`.

fix

Always provide HTML content as bytes. If you have a string, ensure it's encoded correctly before passing it: `html_string.encode('utf-8')`.

affects: All versions

gotchaOlder versions of justext (specifically before v3.0.1) had compatibility issues with newer versions of `lxml`, leading to parsing errors. Similarly, versions before v3.0.0 would fail on Python 3.8+ due to the removal of `cgi.escape`.

fix

Ensure you are using `justext` version 3.0.1 or higher for better `lxml` compatibility, and 3.0.0 or higher for Python 3.8+ environments. Regularly update `justext` to its latest stable release.

affects: < 3.0.1 (lxml compatibility), < 3.0.0 (Python 3.8+)

Errors

Common errors & fixes

AttributeError: 'module' object has no attribute 'get_stoplist'

The `get_stoplist` function is often accessed directly as a top-level function of the `justext` module, but it is actually a method of the `justext` module itself and needs to be called on the module.

fix

Ensure you are calling `get_stoplist` as a method of the `justext` module. The correct way to import and use it is `import justext` and then `justext.get_stoplist('language')`.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x__ in position __: invalid start byte

The HTML content provided to `justext` is not encoded in UTF-8, but the `justext` library (or an upstream `requests` call) is attempting to decode it as such, leading to a decoding failure.

fix

Explicitly specify the correct encoding when fetching the HTML content. If using `requests`, try `response.content.decode('appropriate-encoding')` or `response.encoding = 'appropriate-encoding'` before passing `response.text` or `response.content` to `justext`. You might need to detect the encoding first (e.g., using `chardet`).

requests.exceptions.DecodeError: Received response with content-encoding: gzip, but failed to decode it.

The `requests` library failed to decompress a gzipped HTTP response, indicating a corrupted or incorrectly compressed content stream from the server. This prevents `justext` from receiving valid HTML content.

fix

Handle the decompression error by either setting `decode_content=False` in your `requests.get` call and manually decompressing if possible, or by catching the `DecodeError` and retrying the request. Ensure the server is sending valid gzipped content.

Upgrade

Version history

3.0.2latest on PyPI · released Feb 25, 2025

Audit

Dependencies

lxmlrequiredCore dependency for HTML parsing and DOM manipulation. Changes in lxml can affect justext's parsing capabilities.

Agent activity

3 hits · last 30 days

node

Resources

githubhttps://github.com/miso-belica/jusText ↗

packagehttps://pypi.org/project/justext/ ↗