Install & Compatibility
Where this runs
tested against v1.47.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.960 runs
installs and imports cleanly · install 0.0s · import 0.774s · 47.6MB
glibcpy 3.10–3.960 runs
installs and imports cleanly · install 9.9s · import 0.634s · 49MB
39MB installed
● package 39MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
AsyncScrapeGraphAI
✓ from scrapegraph_py import AsyncScrapeGraphAI
✗ from scrapegraph_py import Client
This quickstart demonstrates how to initialize the ScrapeGraph client using an API key (preferably from an environment variable) and then use the `smartscraper` service to extract structured data from a webpage. It utilizes a Pydantic `BaseModel` to define the desired output schema for robust data validation and type safety. The client should always be closed after use.
import os
from scrapegraph_py import Client
from pydantic import BaseModel, Field
# Set your ScrapeGraph AI API key
# It's recommended to set this as an environment variable: SGAI_API_KEY
# For quick testing, you can pass it directly or use os.environ.get
api_key = os.environ.get('SGAI_API_KEY', 'your_scrapegraph_api_key_here')
if not api_key or api_key == 'your_scrapegraph_api_key_here':
print("Warning: Please set your SGAI_API_KEY environment variable or replace 'your_scrapegraph_api_key_here' with your actual API key.")
exit()
client = Client(api_key=api_key)
class ArticleData(BaseModel):
title: str = Field(description="The article title")
author: str = Field(description="The author's name")
publish_date: str = Field(description="Article publication date")
content: str = Field(description="Main article content")
try:
# Use SmartScraper to extract structured data from a webpage
response = client.smartscraper(
website_url="https://example.com/blog/article-example",
user_prompt="Extract the article information",
output_schema=ArticleData
)
print(f"Title: {response.title}")
print(f"Author: {response.author}")
print(f"Published: {response.publish_date}")
print(f"Content snippet: {response.content[:100]}...")
finally:
# Always close the client connection
client.close()
Debug
Known issues
breakingThe ScrapeGraph ecosystem has seen API changes, particularly with the transition to 'v2 API surface' in related projects like Scrapegraph-ai. While `scrapegraph-py` (the SDK) strives for stability, older code using `ScrapeGraphClient` functions (e.g., `smart_scraper(client, url, prompt)`) may need to be updated to the `Client` class methods (e.g., `client.smartscraper(website_url, user_prompt)`).fixMigrate usage from standalone functions like `smart_scraper(client, ...)` to methods on the `Client` instance, e.g., `client.smartscraper(...)`. Refer to the latest official documentation for current API patterns.
affects: <1.4.x (potentially)
gotchaFailing to set the `SGAI_API_KEY` environment variable or providing an invalid API key will result in authentication errors (HTTP 401 Unauthorized) or 'Insufficient credits' errors. The client will not be able to perform API calls.fixEnsure `SGAI_API_KEY` is set in your environment variables or passed directly to the `Client` constructor: `client = Client(api_key="your-api-key-here")`. Verify the API key's validity on the ScrapeGraphAI Dashboard.
affects: All versions
gotchaWhen using local LLMs with certain functionalities, a 'Model not found, using default token size (8192)' error or an `ImportError: Could not import transformers python package` might occur. This indicates issues with LLM configuration or missing dependencies.fixEnsure `transformers` is installed (`pip install transformers`). If configuring local LLMs, verify the `model_tokens` in your configuration dictionary is correctly set or remove it to use defaults. Consult the ScrapeGraphAI documentation for specific local LLM configurations.
affects: All versions when using local LLMs
gotchaRunning web scraping operations too frequently or without adhering to website policies (e.g., `robots.txt`, terms of service) can lead to IP blocking (HTTP 429 Too Many Requests) or other service unavailability errors (HTTP 500, 503).fixImplement rate limiting and delays between requests. Always check `robots.txt` and a website's terms of service before scraping. The SDK offers automatic retries for certain errors, but manual retry logic with backoff can also be implemented.
affects: All versions
Upgrade
Version history
2.1.0latest on PyPI · released Apr 21, 2026
Audit
Dependencies
pydanticrequiredUsed for structured output with schemas, ensuring type safety.
transformersoptionalRequired for token calculation when using local LLMs; missing can lead to 'Model not found' errors.
playwrightoptionalRecommended for handling dynamic, JavaScript-heavy websites if not using the API's built-in rendering.