Registry / http-networking / crawlerdetect

crawlerdetect

JSON →
library0.3.2pypypi✓ verified 86d ago

CrawlerDetect is a Python library for identifying bots, crawlers, and spiders by analyzing user agent strings. It is a port of the PHP Crawler-Detect library. Current version is 0.3.2, released infrequently.

http-networkingweb-framework
pip install crawlerdetect
github ↗package ↗docs ↗
INSTALL
IMPORT
SIG · CRAWLERDETECT
C
crawlerdetect
http-networkingpythonv0.3.2
Install
1.5s avg
Import
13ms
Disk
16MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.9–3.13
musl
3.9–3.13
Install & Compatibility
Where this runs
tested against v0.3.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.014s · 17.9MB
glibc
py 3.10–3.95 runs
installs and imports cleanly · install 1.5s · import 0.012s · 18MB
16MB installed
● package 16MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

CrawlerDetect
✓ from crawlerdetect import CrawlerDetect
Main class for detecting crawlers.

Basic usage: create a CrawlerDetect instance and call is_crawler with a user agent string.

from crawlerdetect import CrawlerDetect detector = CrawlerDetect() user_agent = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" is_crawler = detector.is_crawler(user_agent) print(f"Is crawler: {is_crawler}") # True
Debug
Known issues
gotchaThe library uses a static list of crawler signatures. If you need custom detection or real-time updates, you must manage the data file yourself.
fix
Fork or extend the library to update the signatures from the PHP source.
affects: all
gotchaUser agent parsing may be case-sensitive. Ensure you pass the exact user agent string without modification.
fix
Pass the raw user agent string as obtained from request headers.
affects: all
deprecatedOlder versions used a different import pattern (e.g., from crawlerdetect import CrawlerDetect as cd). This still works but is deprecated in favor of the direct import.
fix
Use `from crawlerdetect import CrawlerDetect`.
affects: <0.3.0
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'crawlerdetect'
Library not installed or installed in a different environment.
fix
Run `pip install crawlerdetect` in the correct Python environment.
AttributeError: module 'crawlerdetect' has no attribute 'CrawlerDetect'
Using an outdated version or incorrect import statement.
fix
Upgrade to latest version: `pip install --upgrade crawlerdetect`. Then use `from crawlerdetect import CrawlerDetect`.
Upgrade
Version history
0.3.2latest on PyPI · released Jul 9, 2025
Audit
Dependencies

No dependency data recorded yet.

Agent activity
7 hits · last 30 days
node
6
OpenAI (training)
1
Resources
githubhttps://github.com/moskrc/crawlerdetect ↗
packagehttps://pypi.org/project/crawlerdetect/ ↗
docshttps://github.com/moskrc/crawlerdetect ↗
crawlerdetect — pip install crawlerdetect · libregistry