Install & Compatibility
Where this runs
tested against v0.11.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
342MB installed
● package 342MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
ScrapflyClient
✓ from scrapfly import ScrapflyClient
ScrapeConfig
✓ from scrapfly import ScrapeConfig
This quickstart demonstrates how to initialize the Scrapfly client and perform a basic scrape request to a test page. It shows how to enable JavaScript rendering and specify a proxy country. Remember to replace 'YOUR_SCRAPFLY_API_KEY' with your actual key or set the SCRAPFLY_API_KEY environment variable. For HTML parsing with `.selector`, ensure `parsel` or `scrapy` is installed as an extra dependency.
import os
from scrapfly import ScrapflyClient, ScrapeConfig
SCRAPFLY_API_KEY = os.environ.get('SCRAPFLY_API_KEY', 'YOUR_SCRAPFLY_API_KEY')
async def main():
client = ScrapflyClient(key=SCRAPFLY_API_KEY)
try:
result = await client.scrape(ScrapeConfig(url='https://web-scraping.dev/product/1', render_js=True, country='us'))
print(f"Status: {result.status_code}")
print(f"Content length: {len(result.content)} bytes")
# If 'parsel' or 'scrapy' is installed, you can use .selector
# print(f"Product Title: {result.selector.css('h3::text').get()}")
except Exception as e:
print(f"An error occurred: {e}")
finally:
await client.close()
if __name__ == '__main__':
import asyncio
asyncio.run(main())
Debug
Known issues
gotchaAccessing the `ScrapeApiResponse.selector` property for built-in HTML parsing requires installing either `parsel` or `scrapy` as an optional dependency (e.g., `pip install "scrapfly-sdk[parser]"`). Without these, attempting to use `.selector` will result in an `AttributeError`.fixInstall the `parser` or `scrapy` extra: `pip install "scrapfly-sdk[parser]"`
affects: All versions
gotchaHardcoding your Scrapfly API key directly in your code is insecure and inflexible. It's best practice to retrieve it from environment variables or a secure configuration system.fixUse `os.environ.get('SCRAPFLY_API_KEY', 'default_or_error_key')` to load your API key, and set the environment variable `SCRAPFLY_API_KEY`. affects: All versions
gotchaScrapfly API errors (e.g., HTTP 400, 401, 429, 5xx) are encapsulated by `scrapfly.errors.ScrapflyError` subclasses. Incorrect handling or misinterpretation of these can lead to brittle scrapers. Consult the official Scrapfly error documentation for detailed explanations and suggested remedies.fixImplement robust `try-except` blocks for `ScrapflyError` and its subclasses. Refer to the Scrapfly API troubleshooting guide for specific error codes and their meanings.
affects: All versions
breakingWhile not explicitly documented as a breaking change for the Python SDK `0.10.0` specifically, other Scrapfly SDKs (e.g., TypeScript SDK v0.6.9) have undergone parameter renames (e.g., `ephemeral_template` to `extraction_ephemeral_template` in the Extraction API). Always review the official Changelog or release notes for potential API parameter changes when upgrading, especially across minor or major versions.fixBefore upgrading, check the `scrapfly-sdk` GitHub repository's release notes or the official Scrapfly documentation's changelog for any API parameter renames or deprecations.
affects: Future minor/major versions
Errors
Common errors & fixes
AttributeError: 'ScrapeApiResponse' object has no attribute 'selector'
You are trying to use the `.selector` property on a `ScrapeApiResponse` object for HTML parsing, but the necessary optional dependencies (`parsel` or `scrapy`) have not been installed.
fixInstall the `parser` extra: `pip install "scrapfly-sdk[parser]"`. If you are using Scrapy, install `pip install "scrapfly-sdk[scrapy]"` instead.
scrapfly.errors.ScrapflyError: Invalid API key (HTTP 401 Unauthorized)
The Scrapfly API key provided to `ScrapflyClient` is either missing, incorrect, expired, or has insufficient permissions.
fixVerify your API key from your Scrapfly dashboard (https://scrapfly.io/dashboard) and ensure it's correctly passed during client initialization, ideally from an environment variable.
scrapfly.errors.ScrapflyError: Too Many Requests (HTTP 429)
Your Scrapfly account has exceeded its allocated request rate limit or concurrent request limit for the given time period.
fixImplement exponential backoff and retry logic in your scraping code. Review your Scrapfly plan limits or consider upgrading your plan if sustained higher rates are needed.
Scraped content is empty, incomplete, or shows an anti-bot page (no SDK error, but unexpected content).
The target website's anti-bot protection mechanisms successfully identified and blocked the scraping request, or the page content requires JavaScript rendering.
fixIn your `ScrapeConfig`, enable `render_js=True`, `asp=True` (Anti-Scraping Protection), and potentially specify a `proxy_pool='public_residential_pool'` and `country='us'` (or relevant target country).
Upgrade
Version history
0.11.1latest on PyPI · released Jun 14, 2026
Audit
Dependencies
brotlioptionalOptional: For performance improvements (compression) via `[speedups]` extra.
msgpackoptionalOptional: For performance improvements (serialization) via `[speedups]` extra.
parseloptionalOptional: Required for built-in HTML parsing via `ScrapeApiResponse.selector` property, installed with `[parser]` or `[all]` extra.
scrapyoptionalOptional: For Scrapy integration via `[scrapy]` extra. Includes `parsel`.
flaskoptionalOptional: Required for the built-in webhook server via `[webhook-server]` extra.
asynciooptionalOptional: For concurrency features via `[concurrency]` extra, often built-in Python.