Registry / http-networking / trafilatura

trafilatura

JSON →
library2.2.0pypypi✓ verified 25d ago

Trafilatura is a Python and command-line tool designed for gathering text and metadata from the web. It specializes in crawling, scraping, and extracting main content from web pages, supporting various output formats like CSV, JSON, HTML, Markdown, TXT, and XML. The library is actively maintained with frequent releases, offering robust extraction, navigation, and deduplication features.

pip install trafilatura
INSTALL
IMPORT
SIG · TRAFILATURA
T
trafilatura
http-networkingpythonv2.2.0
Install
4.7s avg
Import
3717ms
Disk
79MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v2.2.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.95 runs
installs and imports cleanly · install 0.0s · import 2.174s · 79.7MB
glibc
py 3.103.95 runs
installs and imports cleanly · install 4.7s · import 2.286s · 81MB
79MB installed
● package 79MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

fetch_url
from trafilatura import fetch_url
extract
from trafilatura import extract
bare_extraction
from trafilatura import bare_extraction
result = bare_extraction(html_content, as_dict=True)
`as_dict` argument is deprecated; `bare_extraction()` now returns a `Document` object. Use `.as_dict()` method on the returned object instead.
Document
from trafilatura.settings import Document
Returned by `bare_extraction()`, provides an interface to extracted data including `.as_dict()` method.

This quickstart demonstrates how to fetch a web page and extract its main text content using `trafilatura`. It includes a basic extraction to plain text and an example of extracting structured JSON output with metadata.

from trafilatura import fetch_url, extract import os # Example URL from GitHub blog url = 'https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/' # In a production setting, you might fetch a URL from a variable or a list # For this example, we use a fixed public URL. print(f"Fetching URL: {url}") downloaded_html = fetch_url(url) if downloaded_html: print("Content successfully downloaded. Extracting...") # Extract main content and comments as plain text by default extracted_text = extract(downloaded_html) if extracted_text: print("--- Extracted Text (first 500 chars) ---") print(extracted_text[:500]) print("...") # Example of custom output: JSON with metadata # Note: with_metadata=True is required for metadata inclusion since v1.11.0 print("\n--- Extracting as JSON with metadata ---") extracted_json = extract(downloaded_html, output_format="json", with_metadata=True) if extracted_json: print(extracted_json[:500]) print("...") else: print("Failed to extract content as JSON.") else: print("No text extracted from the downloaded HTML.") else: print(f"Failed to download content from {url}")
trafilatura --version
Debug
Known issues
breakingPython 3.6 and 3.7 are no longer supported. Users must upgrade to Python 3.8 or higher.
fix
Upgrade your Python environment to version 3.8 or newer.
affects: 2.0.0+
breakingThe `bare_extraction()` function now returns an instance of the `Document` class by default. The `as_dict` argument is deprecated.
fix
Access dictionary representation by calling the `.as_dict()` method on the returned `Document` object: `doc = bare_extraction(...); result_dict = doc.as_dict()`.
affects: 2.0.0+
breakingThe `no_fallback` argument in `bare_extraction()` and `extract()` functions has been deprecated.
fix
Use the `fast` argument instead: `extract(html, fast=True)`.
affects: 2.0.0+
breakingThe `decode` argument in `fetch_url()` has been removed.
fix
To get a full response object with control over decoding, use `fetch_response()` directly. `fetch_url()` now seamlessly decodes to a Unicode string.
affects: 2.0.0+
deprecatedMetadata is now skipped by default (`with_metadata=False`).
fix
To include metadata in your output, you must explicitly set `with_metadata=True` in your `extract()` calls or use the `--with-metadata` CLI flag.
affects: 1.11.0+
breakingThe command-line interface (CLI) enforces a fixed list of output formats. The `-out` argument is deprecated.
fix
Use the specified output format options (e.g., `--json`, `--xml`, `--markdown`) instead of `-out`.
affects: 1.12.0+
Upgrade
Version history
2.2.0latest on PyPI · released Jul 31, 2026
Audit
Dependencies
certifirequiredRequired for secure connections.
charset_normalizer>=3.4.0requiredFor character encoding detection.
courlan>=1.3.2requiredUnderlying library for URL management and parsing.
htmldate>=1.9.2requiredFor robust date extraction from HTML.
justext>=3.0.1requiredUsed as a fallback for text extraction.
lxml>=5.3.0requiredCore dependency for HTML parsing and XPath operations.
urllib3<3,>=1.26requiredHTTP client for fetching web pages.
cchardetoptionalOptional: For faster character encoding detection.
langidoptionalOptional: For language detection.
Agent activity
37 hits · last 30 days
node
32
Amazon
1
OpenAI (training)
1
Resources