extruct

library0.18.0pypypi✓ verified 86d ago

Extruct is a Python library for extracting embedded metadata from HTML markup. It currently supports W3C's HTML Microdata, embedded JSON-LD, Microformat (via mf2py), Facebook's Open Graph, experimental RDFa (via rdflib), and Dublin Core Metadata (DC-HTML-2003). The library is actively maintained with its current stable version being 0.18.0.

http-networking data serialization

pip install extruct

github ↗package ↗

INSTALL

IMPORT

SIG · EXTRUCT

extruct

http-networkingpythonv0.18.0

Install

4.8s avg

Import

1499ms

Disk

42MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.18.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.940 runs

installs and imports cleanly · install 0.0s · import 1.564s · 44.1MB

glibc

py 3.10–3.940 runs

installs and imports cleanly · install 4.8s · import 1.435s · 45MB

42MB installed

● package 42MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

extract

✓ from extruct import extract

The primary function for all-in-one metadata extraction.

get_base_url

✓ from w3lib.html import get_base_url

Commonly imported for resolving relative URLs in extracted metadata.

OpenGraphExtractor

✓ from extruct.opengraph import OpenGraphExtractor

Example of importing a specific extractor if only certain formats are needed.

This quickstart fetches HTML content from a URL, determines the base URL for resolving relative paths, and then uses `extruct.extract` to retrieve structured metadata in common formats (JSON-LD, Microdata, Open Graph). The `uniform=True` parameter ensures a consistent output structure for easier processing.

import extruct import requests from w3lib.html import get_base_url import pprint pp = pprint.PrettyPrinter(indent=2) # Replace with a real URL to test url = 'http://quotes.toscrape.com/scroll' r = requests.get(url) base_url = get_base_url(r.text, r.url) data = extruct.extract(r.text, base_url=base_url, uniform=True, syntaxes=['json-ld', 'microdata', 'opengraph']) pp.pprint(data)

extruct --version

Debug

Known issues

breakingVersions of `extruct` prior to 0.18.0 might encounter `ImportError: cannot import name '_ElementStringResult' from 'lxml.etree'` when used with `lxml` versions 5.1.0 or higher due to internal API changes in `lxml`.

fix

Upgrade `extruct` to version 0.18.0 or newer. If an upgrade is not feasible, temporarily pin `lxml` to a version less than 5.1.0 (e.g., `pip install lxml<5.1.0`).

affects: <0.18.0

gotchaThe output structure of `extruct` can be inconsistent for certain metadata types, sometimes returning a list of dictionaries and other times a single dictionary, which can lead to `TypeError` or `IndexError` if not handled carefully in post-processing.

fix

Use the `uniform=True` parameter in `extruct.extract()` to ensure a more consistent output structure (e.g., always a list). Always check the type of the returned data before attempting to access elements by index or key.

affects: All versions

gotchaExtracting all supported syntaxes from very large or complex HTML documents can be memory-intensive and slow. By default, `extruct.extract()` attempts all formats.

fix

To optimize performance and resource usage, specify only the required syntaxes using the `syntaxes` parameter (e.g., `syntaxes=['json-ld', 'opengraph']`).

affects: All versions

gotchaThe command-line tool `extruct` (e.g., `extruct 'http://example.com'`) requires the `requests` library, which is an optional dependency and not installed by default with a basic `pip install extruct`.

fix

Install `extruct` with the `cli` extra to include `requests`: `pip install 'extruct[cli]'`.

affects: All versions

Errors

Common errors & fixes

ImportError: cannot import name '_ElementStringResult' from 'lxml.etree'

An incompatibility between older `extruct` versions (<0.18.0) and `lxml` versions 5.1.0 or newer.

fix

Update `extruct` to version 0.18.0 or later. If updating `extruct` is not possible, downgrade `lxml` to a version prior to 5.1.0 (e.g., `pip install lxml==5.0.1`).

Empty dictionary or unexpected missing metadata in extruct output.

The target HTML either does not contain metadata in the formats `extruct` supports, or relative URLs were not resolved because `base_url` was omitted.

fix

First, inspect the source HTML for the presence of Microdata, JSON-LD, Open Graph, etc. Second, always provide the `base_url` parameter to `extruct.extract(html_string, base_url=actual_url)` to ensure proper resolution of relative URLs and images.

ModuleNotFoundError: No module named 'requests' when running the `extruct` command-line tool.

The `requests` library, which the command-line interface uses to fetch web pages, is an optional dependency and not installed by default.

fix

Install `extruct` with its command-line interface dependencies using `pip install 'extruct[cli]'`.

Upgrade

Version history

0.18.0latest on PyPI · released Nov 8, 2024

Audit

Dependencies

lxmlrequiredCore dependency for HTML parsing.

w3librequiredUsed for HTML utilities like getting the base URL.

mf2pyrequiredRequired for Microformat extraction.

rdflibrequiredRequired for experimental RDFa extraction.

jstylesonrequiredUsed for robust JSON-LD parsing.

lxml-html-cleanrequiredUsed for cleaning HTML before parsing.

pyrdfa3requiredRelated to RDFa parsing.

requestsoptionalOptional dependency for the command-line interface to fetch URLs.

Agent activity

24 hits · last 30 days

node

Amazon

OpenAI (training)

Resources

githubhttps://github.com/scrapinghub/extruct ↗

packagehttps://pypi.org/project/extruct/ ↗