icrawler

library0.6.10pypypi✓ verified 83d ago

icrawler is a multi-thread crawler framework with many built-in image crawlers for search engines like Google, Bing, Baidu, etc. Current version is 0.6.10, released in November 2022. Release cadence is low, last update in 2022.

analytics

pip install icrawler

INSTALL

IMPORT

SIG · ICRAWLER

icrawler

analyticspythonv0.6.10

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

ImageCrawler

✓ from icrawler.builtin import GoogleImageCrawler

✗ from icrawler import ImageCrawler

ImageCrawler is a base class; built-in crawlers are in icrawler.builtin

BingImageCrawler

✓ from icrawler.builtin import BingImageCrawler

BaiduImageCrawler

✓ from icrawler.builtin import BaiduImageCrawler

UrlListCrawler

✓ from icrawler.builtin import UrlListCrawler

Downloads 10 images of cats from Google Image Search into 'downloads' directory.

from icrawler.builtin import GoogleImageCrawler crawler = GoogleImageCrawler(storage={'root_dir': 'downloads'}) crawler.crawl(keyword='cat', max_num=10)

Debug

Known issues

gotchaCrawlers may not work due to search engine anti-bot measures (CAPTCHA, IP blocking). Consider using official APIs instead.

fix

Use legal image APIs or scrape responsibly with delays/user-agent rotation.

affects: all

gotchaImage download may fail silently if URLs are inaccessible or invalid. Check logs for errors.

fix

Enable logging: import logging; logging.basicConfig(level=logging.INFO)

affects: all

deprecatedPython 2 support via 'six' is deprecated. Use Python >=3.7.

fix

Upgrade Python to 3.7+

affects: 0.6.10

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'icrawler'

icrawler not installed or installed in wrong environment.

fix

Run: pip install icrawler

ImportError: cannot import name 'GoogleImageCrawler' from 'icrawler'

Incorrect import path (e.g., from icrawler import GoogleImageCrawler).

fix

Use: from icrawler.builtin import GoogleImageCrawler

AttributeError: 'NoneType' object has no attribute 'group'

Parser failed to extract image URLs, often due to website changes or network issues.

fix

Ensure network connectivity and update the library if available.

Upgrade

Version history

0.6.10latest on PyPI · released Feb 11, 2025

Audit

Dependencies

requestsrequiredUsed for HTTP requests

PillowrequiredImage processing and validation

sixrequiredPython 2/3 compatibility (legacy)

Agent activity

27 hits · last 30 days

node

OpenAI (training)

Resources

No resource links recorded.