Install & Compatibility
Where this runs
tested against v2.4.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.110s · 17.9MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 1.5s · import 0.094s · 18MB
16MB installed
● package 16MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
canonicalize_url
✓ from w3lib.url import canonicalize_url
remove_tags
✓ from w3lib.html import remove_tags
headers_raw_to_dict
✓ from w3lib.http import headers_raw_to_dict
str_to_unicode
✓ w3lib.util.str_to_unicode is removed in v2.3.0+
✗ from w3lib.util import str_to_unicode
Functions `str_to_unicode`, `to_native_str`, `unicode_to_str` were removed in v2.3.0. Use standard Python string encoding/decoding methods instead.
This quickstart demonstrates basic usage of w3lib for URL canonicalization, HTML tag removal, and HTTP header parsing.
from w3lib.url import canonicalize_url
from w3lib.html import remove_tags
from w3lib.http import headers_raw_to_dict
# URL canonicalization
url = "http://example.com/path/../foo.html?a=1&b=2#frag"
canonical_url = canonicalize_url(url)
print(f"Canonical URL: {canonical_url}")
# HTML tag removal
html_content = "<div>Hello <b>world</b>!</div>"
clean_text = remove_tags(html_content)
print(f"Clean text: {clean_text}")
# HTTP headers parsing
raw_headers = b"Content-Type: text/html\r\nUser-Agent: my-spider/1.0"
parsed_headers = headers_raw_to_dict(raw_headers)
print(f"Parsed Headers: {parsed_headers}")
Debug
Known issues
breakingPython 2 support was dropped in v2.0.1. Users must migrate to Python 3.6+.fixUpgrade your Python environment to 3.6 or later. For current versions (2.4.x), Python 3.10+ is required.
affects: <2.0.1
breakingSeveral Python versions have had their support dropped in recent releases (e.g., 3.7 in v2.2.0, 3.8 in v2.3.0, 3.9 in v2.4.0). Ensure your Python environment meets the minimum requirement for the installed w3lib version.fixUpgrade your Python environment to Python 3.10+ for w3lib versions 2.4.x and newer.
affects: >=2.2.0
breakingThe utility functions `w3lib.util.str_to_unicode`, `w3lib.util.to_native_str`, and `w3lib.util.unicode_to_str` were removed in v2.3.0. These were deprecated in v2.0.0.fixReplace calls to these functions with standard Python string encoding and decoding methods (e.g., `.encode()` and `.decode()`).
affects: >=2.3.0
gotchaThe behavior of `w3lib.url.canonicalize_url` and `w3lib.url.safe_url_string` has changed regarding how they handle `%23` and userinfo components, potentially affecting URL fingerprinting or comparisons.fixReview your application's reliance on the exact output of these functions, especially if URL fingerprinting or strict URL comparisons are critical. Test thoroughly after upgrading.
affects: >=2.0.1, >=2.2.1
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'w3lib'
The 'w3lib' library has not been installed in your current Python environment.
ImportError: cannot import name 'url_query_cleaner' from 'w3lib'
Functions like 'url_query_cleaner' are located within specific submodules (e.g., 'w3lib.url', 'w3lib.html'), not directly in the top-level 'w3lib' package.
fixfrom w3lib.url import url_query_cleaner
TypeError: expected string or bytes-like object, got int
A 'w3lib' function that expects a string or bytes-like object (e.g., HTML content or a URL) received an input of an incompatible type, such as an integer.
fixfrom w3lib.html import remove_tags
html_content = 123 # Example non-string input
cleaned_content = remove_tags(str(html_content)) # Convert to string before passing
Upgrade
Version history
2.4.1latest on PyPI · released Mar 20, 2026
Audit
Dependencies
No dependency data recorded yet.