protego

library0.6.2pypypi✓ verified 27d ago

Protego is a pure-Python robots.txt parser with support for modern conventions like those defined by Google. As of version 0.6.0, it actively supports Python 3.10 and newer, with regular updates aligning with new Python releases. It is widely used for web scraping and compliance checking.

http-networking data web-framework

pip install protego

github ↗package ↗changelog ↗

INSTALL

IMPORT

SIG · PROTEGO

protego

http-networkingpythonv0.6.2

Install

1.5s avg

Import

45ms

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.6.2 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.048s · 17.8MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.5s · import 0.042s · 18MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Protego

✓ from protego import Protego

Initialize the parser with robots.txt content and check URL access permissions for a specific user agent, retrieve crawl delay, and sitemaps.

from protego import Protego robotstxt_content = """ User-agent: * Disallow: /admin/ Allow: /admin/login Crawl-delay: 5 Sitemap: http://example.com/sitemap.xml """ rp = Protego.parse(robotstxt_content) # Check if a URL can be fetched by a user agent can_fetch_admin = rp.can_fetch("http://example.com/admin/settings", "mybot") can_fetch_login = rp.can_fetch("http://example.com/admin/login", "mybot") print(f"Can 'mybot' fetch /admin/settings? {can_fetch_admin}") print(f"Can 'mybot' fetch /admin/login? {can_fetch_login}") print(f"Crawl delay for 'mybot': {rp.crawl_delay('mybot')} seconds") print(f"Sitemaps: {list(rp.sitemaps)}")

Debug

Known issues

breakingVersion 0.6.0 dropped official support for Python 3.9 and PyPy 3.10. Users on these Python versions should use Protego 0.5.x or upgrade their Python environment.

fix

Upgrade to Python 3.10+ or pin Protego to a version <0.6.0 (e.g., `protego<0.6.0`).

affects: >=0.6.0

breakingVersion 0.4.0 dropped official support for Python 3.8.

fix

Upgrade to Python 3.9+ or pin Protego to a version <0.4.0.

affects: >=0.4.0

breakingVersion 0.3.0 dropped support for Python 2.7, 3.5, 3.6, and 3.7. The `six` dependency was also removed in this version, making it Python 3 only.

fix

Upgrade to Python 3.8+ and remove `six` if it was only a Protego dependency. Pin Protego to a version <0.3.0 for older Python environments.

affects: >=0.3.0

gotchaIn Protego 0.3.0 and later, `Protego.parse()` will raise a `ValueError` if the `robotstxt_body` argument is not a string.

fix

Ensure that the `robotstxt_body` passed to `Protego.parse()` is always a string.

affects: >=0.3.0

gotchaVersion 0.1.16 fixed an issue where absolute URLs in `Allow` and `Disallow` directives were incorrectly parsed, ignoring their protocol and netloc. Older versions might misinterpret these directives, leading to incorrect access decisions.

fix

Upgrade to Protego 0.1.16 or newer to ensure correct interpretation of absolute URLs in `robots.txt` directives.

affects: <0.1.16

gotchaVersion 0.5.0 restructured the internal code from a single `protego.py` file into multiple modules. While the public API `from protego import Protego` remains stable, direct imports of internal modules (if any were used) would have broken.

fix

Avoid importing internal modules; rely only on the documented public API (e.g., `from protego import Protego`).

affects: >=0.5.0

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'protego'

The `protego` library has not been installed in the current Python environment.

fix

Install the library using pip: `pip install protego`

ValueError: content is not a string

The `Protego.parse()` method expects its input (`robotstxt_body`) to be a string, but it received a non-string type (e.g., bytes). This behavior was explicitly introduced in Protego 0.6.0 and newer versions.

fix

Ensure that the `robotstxt_body` passed to `Protego.parse()` is decoded into a string (e.g., UTF-8) before parsing. Example: `Protego.parse(response.text)` if using `requests`, or `Protego.parse(robotstxt_bytes.decode('utf-8'))`.

AttributeError: module 'protego' has no attribute 'parse'

This error occurs when attempting to call `parse()` directly on the `protego` module (e.g., `protego.parse(...)`) instead of on an instance of the `Protego` class. The `parse` method is a static method of the `Protego` class.

fix

Import the `Protego` class and call `parse()` as a static method on the class: `from protego import Protego` then `robots = Protego.parse(robots_txt_content)`.

Upgrade

Version history

0.6.2latest on PyPI · released Jun 25, 2026

Audit

Dependencies

No dependency data recorded yet.

Agent activity

20 hits · last 30 days

node

Resources

githubhttps://github.com/scrapy/protego ↗

packagehttps://pypi.org/project/protego/ ↗

changeloghttps://github.com/scrapy/protego/blob/master/CHANGELOG.rst ↗