Registry / http-networking / scrapingbee

scrapingbee

library2.0.2pypypi✓ verified 85d ago

ScrapingBee is a web scraping API that handles headless browsers and rotates proxies for you. The Python SDK simplifies interaction with this API, offering features like JavaScript rendering, proxy rotation, AI-powered data extraction, and screenshot capabilities. It is currently at version 2.0.2 and receives regular updates, focusing on reliability and new API features.

http-networking data

pip install scrapingbee

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · SCRAPINGBEE

scrapingbee

http-networkingpythonv2.0.2

Install

2.1s avg

Import

616ms

Disk

19MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.0.2 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

installs and imports cleanly · install 0.0s · import 0.652s · 21.2MB

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 2.1s · import 0.580s · 22MB

19MB installed

● package 19MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

ScrapingBeeClient

✓ from scrapingbee import ScrapingBeeClient

This quickstart initializes the ScrapingBee client with an API key (preferably from an environment variable) and sends a GET request to a URL. It demonstrates using `extract_rules` to automatically parse specific data (title, subtitle, article headings) from the page into a JSON format.

import os from scrapingbee import ScrapingBeeClient # It's highly recommended to store your API key in an environment variable api_key = os.environ.get('SCRAPINGBEE_API_KEY', 'YOUR_API_KEY') if api_key == 'YOUR_API_KEY': print("Warning: Replace 'YOUR_API_KEY' or set the SCRAPINGBEE_API_KEY environment variable.") client = ScrapingBeeClient(api_key=api_key) url_to_scrape = 'https://www.scrapingbee.com/blog/' try: response = client.get( url_to_scrape, params={ 'render_js': True, # Set to False to save credits if JavaScript rendering is not needed 'extract_rules': { 'title': 'h1', 'subtitle': '#subtitle', 'articles': {'selector': 'article h2 a', 'type': 'list', 'output': 'text'} } } ) if response.ok: # If extract_rules are used, the content is usually JSON if response.headers.get('content-type') == 'application/json': import json data = json.loads(response.content) print(json.dumps(data, indent=2)) else: # Otherwise, it's the raw HTML print(response.text[:500]) # Print first 500 characters of HTML else: print(f"Failed to scrape {url_to_scrape}: Status {response.status_code}, Content: {response.text[:200]}") except Exception as e: print(f"An error occurred: {e}")

Debug

Known issues

breakingVersion 2.0.0 introduced a fix for URL encoding of parameters. While intended as a correction, this might alter the behavior for existing users who might have implicitly relied on or worked around previous (potentially incorrect) encoding, leading to different request URLs or parameter interpretation. [cite: GitHub release v2.0.0]

fix

Review your `params` dictionary and the target URLs to ensure they are correctly interpreted by the API's new encoding logic. Test existing scraping jobs thoroughly.

affects: >=2.0.0

breakingPython 3.6 support was officially dropped in version 1.2.0. Users running on Python 3.6 will encounter issues or be unable to upgrade past v1.1.8. [cite: GitHub release v1.2.0]

fix

Upgrade your Python environment to version 3.8 or newer. Python 3.6 has reached its end-of-life and is no longer supported by the Python core team.

affects: >=1.2.0

gotchaHardcoding your ScrapingBee API key directly into your scripts is a security risk. It should be stored securely, ideally in an environment variable.

fix

Retrieve your API key from an environment variable (e.g., `os.environ.get('SCRAPINGBEE_API_KEY')`) rather than embedding it directly in your code. Ensure this variable is set in your deployment environment.

affects: All

gotchaBy default, `render_js` is set to `True` for `client.get()` requests, which means JavaScript is executed and consumes 5 credits per request. For simple static HTML pages, this can unnecessarily increase credit usage.

fix

For pages that do not require JavaScript execution, set `render_js=False` in your `params` dictionary to save credits. E.g., `client.get(url, params={'render_js': False})`.

affects: All

gotchaScrapingBee plans have limits on concurrent requests. Exceeding this limit can lead to requests being queued or failing.

fix

Implement proper concurrency management in your scraping logic (e.g., using `concurrent.futures.ThreadPoolExecutor` in Python) to respect your plan's concurrent request limit.

affects: All

gotchaWhile `extract_rules` are powerful, like any CSS/XPath selectors, they can break if the target website's HTML structure changes. The `v2.0.2` release specifically fixed handling of AI extract rules, indicating this is an area where issues can arise. [cite: GitHub release v2.0.2, 20]

fix

Regularly monitor your scrapers and adjust `extract_rules` as needed when website layouts change. Consider using AI-powered extraction or a combination of robust selectors and post-processing for critical data.

affects: All

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'scrapingbee'

The Python package for ScrapingBee was not installed, or was installed under a different name, preventing the 'scrapingbee' module from being found.

fix

Install the official ScrapingBee Python SDK using pip: `pip install scrapingbee-python-sdk`

ScrapingBeeAPIError: 403 Forbidden - Invalid API Key

The API key provided to the ScrapingBee client is either incorrect, expired, or not authorized for the requested operation.

fix

Ensure you are using a valid and active API key from your ScrapingBee dashboard and pass it correctly: `client = ScrapingBeeClient(api_key='YOUR_VALID_API_KEY')`

requests.exceptions.Timeout

The request to the ScrapingBee API or the target URL took longer than the specified timeout duration, causing the client-side connection to close.

fix

Increase the timeout duration for your requests, both for the ScrapingBee API and potentially your local client: `response = client.get('https://example.com', params={'timeout': 60000}, timeout=45)`

ScrapingBeeAPIError: 400 Bad Request - 'javascript_scenario_steps must be an array'

The `javascript_scenario_steps` parameter was provided as a single dictionary instead of a list of dictionaries, which is the required format for the ScrapingBee API.

fix

Always pass `javascript_scenario_steps` as a list of dictionaries, even if it contains only one step: `params={'javascript_scenario': True, 'javascript_scenario_steps': [{'click': '.some-button'}]}`

ImportError: cannot import name 'Client' from 'scrapingbee'

The developer attempted to import an incorrect class name 'Client' instead of the actual class name 'ScrapingBeeClient' from the 'scrapingbee' module.

fix

Use the correct class name 'ScrapingBeeClient' in your import statement: `from scrapingbee import ScrapingBeeClient`

Upgrade

Version history

2.0.2latest on PyPI · released Oct 2, 2025

Audit

Dependencies

requestsrequiredThe ScrapingBee Python SDK is a wrapper around the requests library, using it for HTTP communication.

Agent activity

18 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/scrapingbee/scrapingbee-python ↗

packagehttps://pypi.org/project/scrapingbee/ ↗

homepagehttps://www.scrapingbee.com ↗