Registry / llm-agents / maincontentextractor

maincontentextractor

library0.0.4pypypi✓ verified 23d ago

MainContentExtractor is a Python library designed to extract the core content from HTML documents. It aims to address limitations found in other extraction tools, such as the inability to output clean HTML directly. The library is useful for LLM-related tasks and for feeding data into frameworks like LangChain and LlamaIndex by providing output in HTML, Text, or Markdown formats. It is currently at version 0.0.4, with a relatively active development cadence.

llm-agents web-framework data

pip install MainContentExtractor

github ↗package ↗

INSTALL

IMPORT

SIG · MAINCONTENTEXTRACT

maincontentextractor

llm-agentspythonv0.0.4

Install

5.2s avg

Import

4350ms

Disk

81MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.0.4 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 2.562s · 81.4MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 5.2s · import 2.658s · 82MB

81MB installed

● package 81MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

MainContentExtractor

✓ from main_content_extractor import MainContentExtractor

✗ from MainContentExtractor import MainContentExtractor

The PyPI package name is `MainContentExtractor` (capitalized), but the Python module and class name use `main_content_extractor` (lowercase with underscore). Ensure correct casing for import statements.

This quickstart demonstrates how to extract the main content from an HTML string using MainContentExtractor. It shows output in HTML, Markdown, and plain text formats. If fetching HTML from a URL, ensure `requests` is installed (`pip install requests`).

import requests from main_content_extractor import MainContentExtractor # Example HTML content (or fetch from a URL) html_content = """ <html> <head><title>Example Page</title></head> <body> <header>Navigation Bar</header> <main> <h1>Important Article Title</h1> <p>This is the main content paragraph.</p> <p>Another paragraph with <a href="#">a link</a> inside.</p> </main> <footer>Footer content</footer> </body> </html> """ # Or, fetch from a URL (requires 'requests') # url = "https://www.example.com" # response = requests.get(url) # response.encoding = 'utf-8' # html_content = response.text # Extract main content as HTML extracted_html = MainContentExtractor.extract(html_content) print("--- Extracted HTML ---") print(extracted_html) # Extract main content as Markdown extracted_markdown = MainContentExtractor.extract(html_content, output_format="markdown") print("\n--- Extracted Markdown ---") print(extracted_markdown) # Extract main content as plain text extracted_text = MainContentExtractor.extract(html_content, output_format="text") print("\n--- Extracted Text ---") print(extracted_text)

Debug

Known issues

gotchaThe library internally uses `trafilatura` and converts its XML output to HTML. This conversion is described as irreversible and may not perfectly match the original HTML structure.

fix

Be aware that the output HTML might not be an exact replica of the original main content. Validate the extracted output against your specific use case, especially if pixel-perfect fidelity to the original HTML is required.

affects: 0.0.1 - 0.0.4

gotchaThere can be confusion between the PyPI package name (`MainContentExtractor` - capitalized) and the Python module name for import (`main_content_extractor` - lowercase with underscores). A `ModuleNotFoundError` will occur if the import statement uses the incorrect casing or formatting.

fix

Always use `pip install MainContentExtractor` for installation and `from main_content_extractor import MainContentExtractor` for importing. Pay close attention to casing and underscores.

affects: 0.0.1 - 0.0.4

breakingGiven the library's early stage (version 0.0.4), API stability is not guaranteed. Minor version updates may introduce breaking changes or significant modifications to the API without extensive deprecation warnings.

fix

Pin your project's dependency to a specific patch version (e.g., `MainContentExtractor==0.0.4`) and thoroughly test your application after any updates. Monitor the GitHub repository for release notes and changes.

affects: < 1.0.0

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'main_content_extractor'

The Python interpreter cannot find the 'main_content_extractor' module because it was either not installed, or there's a casing mismatch between how the package was installed (often as 'MainContentExtractor') and how it's being imported.

fix

Ensure the package is installed with the correct casing using `pip install MainContentExtractor`. Then, verify your Python import statement matches the installed module, for example: `from main_content_extractor import MainContentExtractor`.

ToolException Details: Error running Browser Use Agent: No module named 'MainContentExtractor'

This error, often encountered when using `maincontentextractor` as a dependency (e.g., within `browser-use`), indicates that the `MainContentExtractor` module cannot be found, typically due to incorrect installation casing or a missing package in the environment.

fix

Confirm `maincontentextractor` is installed correctly, specifically using the casing `pip install MainContentExtractor`. If using a virtual environment, ensure it's activated and the package is present.

ImportError: lxml.html.clean module is now a separate project lxml_html_clean. Install lxml[html_clean] or lxml_html_clean directly.

A dependency of `maincontentextractor` (such as `justext`) attempts to import `lxml.html.clean`, but this module has been moved to a separate package in recent `lxml` versions, causing a runtime import failure.

fix

Install the required sub-dependency by running: `pip install lxml_html_clean`.

Upgrade

Version history

0.0.4latest on PyPI · released Dec 10, 2023

Audit

Dependencies

beautifulsoup4requiredUsed for HTML parsing and manipulation.

html2textrequiredUsed for converting HTML to Markdown or plain text.

trafilaturarequiredThe core main content extraction relies on trafilatura internally.

Agent activity

14 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/HawkClaws/main_content_extractor ↗

packagehttps://pypi.org/project/maincontentextractor/ ↗