Registry / http-networking / spider-client

spider-client

library0.1.88pypypi✓ verified 79d ago

The `spider-client` is a Python SDK for integrating with the Spider Cloud API, providing tools for web scraping, large-scale crawling, link extraction, and taking screenshots. It is designed to efficiently collect data, often formatted for compatibility with Language Models (LLMs), leveraging a Rust-based engine optimized for AI that supports concurrent operations, streaming, and headless Chrome rendering. The library is actively maintained, with frequent updates, and the current version is 0.1.88.

http-networking ai-ml llm-agents data

pip install spider-client

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · SPIDER-CLIENT

spider-client

http-networkingpythonv0.1.88

Install

4.6s avg

Import

—

Disk

30MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.88 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

installs and imports cleanly · install 0.0s · import 0.000s · 30.7MB

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 4.6s · import 0.000s · 33MB

30MB installed

● package 30MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Spider

✓ from spider import Spider

✗ from spider_client import Spider

AsyncSpider

✓ from spider import AsyncSpider

spider

✓ from spider import spider

This quickstart initializes the Spider client and performs a basic URL scrape. It demonstrates how to configure the API key, either via an environment variable or direct instantiation, and handles a simple scraping operation. Obtain your API key from spider.cloud.

import os from spider_client import Spider # Retrieve API key from environment variable or replace with your actual key # Get an API key from https://spider.cloud api_key = os.environ.get('SPIDER_API_KEY', 'YOUR_SPIDER_API_KEY') if not api_key or api_key == 'YOUR_SPIDER_API_KEY': print("WARNING: SPIDER_API_KEY not set. Please set it as an environment variable or pass to Spider(api_key=...).\nSkipping API call.") else: app = Spider(api_key=api_key) url_to_scrape = 'https://example.com' try: scraped_data = app.scrape_url(url_to_scrape) print(f"Successfully scraped data from {url_to_scrape}:") print(scraped_data) except Exception as e: print(f"An error occurred during scraping: {e}")

spider-client --version

Debug

Known issues

breakingThe `v0.1.37` release included a fix that 'removed pipe operator' (`fix(python): removed pipe operator`). This likely pertains to changes in type hinting syntax or internal handling of union types, which might affect compatibility with specific older Python versions or code relying on a previous internal implementation.

fix

Ensure your Python environment is up-to-date (preferably Python 3.9+ for native pipe operator syntax) and review any custom type hint definitions that might conflict with the change. If issues persist, check the official GitHub for detailed migration guides.

affects: <0.1.37

gotchaThe library requires an API key for authentication with the Spider Cloud API. Requests without a valid API key will fail with authentication errors.

fix

Obtain an API key from spider.cloud and set it as an environment variable `SPIDER_API_KEY` or pass it directly to the `Spider` constructor: `app = Spider(api_key='YOUR_API_KEY')`.

affects: All versions

gotchaWhen dealing with large JSON responses or streaming data, direct `json.loads()` on the entire response might lead to `json.JSONDecodeError` due to incomplete data or excessive memory usage.

fix

Utilize the `stream=True` parameter in methods like `crawl_url` and `scrape_url`, and process the response iteratively. The library internally uses `ijson` for efficient streaming. Follow examples for processing chunks or streaming directly.

affects: All versions

Upgrade

Version history

0.1.88latest on PyPI · released Mar 20, 2026

Audit

Dependencies

ijsonrequiredUsed for iterative parsing of large JSON streams, added in v0.1.37 to improve handling of substantial data payloads.

Agent activity

36 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/spider-rs/spider-clients ↗

packagehttps://pypi.org/project/spider-client/ ↗

homepagehttps://spider.cloud ↗