courlan

library1.4.0pypypi✓ verified 25d ago

Courlan (version 1.3.2) is a Python library designed to clean, filter, and sample URLs, optimizing data collection workflows. It includes features for spam detection, content type filtering, and language identification. The library maintains an active development pace with minor releases typically every few months.

http-networking data

pip install courlan

github ↗package ↗

INSTALL

IMPORT

SIG · COURLAN

courlan

http-networkingpythonv1.4.0

Install

2.3s avg

Import

366ms

Disk

51MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.4.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.382s · 53MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 2.3s · import 0.350s · 54MB

51MB installed

● package 51MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

clean_url

✓ from courlan import clean_url

extract_links

✓ from courlan import extract_links

UrlStore

✓ from courlan import UrlStore

This quickstart demonstrates basic URL cleaning, link extraction from HTML, and usage of the `UrlStore` for managing visited and unvisited URLs. Link extraction often benefits from a `base_url` to resolve relative links, and the `UrlStore` provides efficient tracking for web crawling applications.

from courlan import clean_url, extract_links, UrlStore # Example 1: Clean a URL raw_url = 'http://www.Example.com/path/?query=value#fragment' cleaned_url = clean_url(raw_url) print(f"Cleaned URL: {cleaned_url}") # Example 2: Extract links from HTML (requires lxml) html_content = """ <html> <body> <a href="/relative/path">Relative Link</a> <a href="https://example.org/absolute">Absolute Link</a> <a href="http://invalid.com?utm_source=foo">Tracker Link</a> </body> </html> """ extracted_links = extract_links( html_content, url='https://example.com/base', deduplicate=True, with_fragment=False, with_query=False, original_url_and_query=False ) print(f"Extracted links: {list(extracted_links.keys())}") # Example 3: Using UrlStore store = UrlStore() store.add('https://example.com/page1') store.add('https://example.org/page2') print(f"URLs in store: {store.size}") # Mark a URL as visited store.visit('https://example.com/page1') unvisited_urls = store.get_unvisited_urls() print(f"Unvisited URLs: {list(unvisited_urls)}")

Debug

Known issues

breakingPython 3.6 and 3.7 support was officially dropped with version 1.3.0. Users on these older Python versions must upgrade to Python 3.8 or newer to use courlan 1.3.0 and later.

fix

Upgrade your Python environment to version 3.8 or higher.

affects: >=1.3.0

breakingThe `timelimit` parameter was entirely removed from the `UrlStore.get_download_urls()` method in version 1.3.2. For other `UrlStore` methods, the parameter was renamed from `timelimit` to `time_limit` in version 1.1.0, with the old name being deprecated in 1.2.0.

fix

For `UrlStore.get_download_urls()`, remove the `timelimit` parameter. For other `UrlStore` methods, rename `timelimit` to `time_limit`.

affects: >=1.3.2 for `get_download_urls()`, >=1.1.0 for other `UrlStore` methods

deprecatedThe `base_url` parameter in `extract_links()` was deprecated in version 1.3.1 and is scheduled for removal. While it currently still works, it's advised to avoid its use.

fix

The intended replacement for handling relative URLs is often to clean them after extraction using `clean_url` with an explicit base, or to ensure the input `url` parameter to `extract_links` provides a sufficient base for resolution. Consult the latest documentation for alternatives as the parameter's removal approaches.

affects: >=1.3.1

gotchaStarting with version 1.3.1, `UrlStore` compression using `bz2` or `zlib` is optional. If you attempt to use these compression types without installing the respective Python packages (`python-bzip2` or `zlib-python`), `UrlStore` will raise an error or fall back to an uncompressed state.

fix

Explicitly install the required packages: `pip install python-bzip2` for bzip2 compression, or ensure zlib development libraries are available for zlib-python (often built-in with Python).

affects: >=1.3.1

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'courlan.core'

Users might attempt to import specific submodules directly (e.g., `courlan.core`) which are not exposed at the top level or are not intended for direct import, leading to a `ModuleNotFoundError`.

fix

Most commonly used functions are available directly from the top-level `courlan` package. If a function is in a submodule, import it from `courlan` directly, or check the documentation for the correct import path. For example, `from courlan import get_base_url` instead of trying to guess the submodule.

AttributeError: module 'courlan' has no attribute 'filter_url'

This error occurs when a user tries to call a function (like `filter_url`) that either doesn't exist, has been renamed, or is part of a different object/class within the `courlan` library.

fix

Consult the `courlan` documentation or source code to confirm the correct function name and its location. For example, `courlan` offers `filter_links()` for link filtering.

courlan cache clear

While not a direct error message, users search for this when `courlan` exhibits unexpected behavior, such as processing URLs inconsistently or not applying new filtering rules, often due to stale internal caches.

fix

To reset the internal cache and resolve inconsistencies, import `clear_caches` from `courlan.meta` and call it: `from courlan.meta import clear_caches; clear_caches()`

Upgrade

Version history

1.4.0latest on PyPI · released Jun 1, 2026

Audit

Dependencies

lxmlrequiredRequired for parsing HTML content to extract links efficiently.

babelrequiredUsed for locale information, particularly for language filtering and identification.

python-bzip2optionalEnables bzip2 compression for UrlStore if desired. Not installed by default.

zlib-pythonoptionalEnables zlib compression for UrlStore if desired. Not installed by default.

Agent activity

21 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/adbar/courlan ↗

packagehttps://pypi.org/project/courlan/ ↗