Registry / web-framework / scrapy-playwright

scrapy-playwright

library0.0.47pypypiunverified

Scrapy Playwright is a Scrapy Download Handler that integrates Playwright for Python, enabling Scrapy spiders to effectively scrape dynamic web pages that rely heavily on JavaScript rendering. It allows for browser automation within the Scrapy framework, facilitating interaction with complex web elements, while maintaining Scrapy's efficient crawling and scheduling model. The library is actively maintained with frequent releases, currently at version 0.0.46.

web-framework http-networking testing

pip install scrapy-playwright playwright

github ↗package ↗

INSTALL

IMPORT

SIG · SCRAPY-PLAYWRIGHT

scrapy-playwright

web-frameworkpythonv0.0.47

Install

9.4s avg

Import

1763ms

Disk

235MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.0.47 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

build_error

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 9.4s · import 1.763s · 232MB

235MB installed

● package 235MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

ScrapyPlaywrightDownloadHandler

✓ from scrapy_playwright.handler import ScrapyPlaywrightDownloadHandler

Used in settings.py to enable Playwright for HTTP/HTTPS requests.

PageMethod

✓ from scrapy_playwright.page import PageMethod

Used to define actions (e.g., click, wait_for_selector) within a Playwright request's meta field.

PlaywrightRequest

✓ from scrapy_playwright.request import PlaywrightRequest

✗ from scrapy_playwright.page import PlaywrightRequest

While Request objects are usually from 'scrapy', some examples might mistakenly use 'page' or an old path for Playwright-specific request objects. The standard approach is to use `scrapy.Request` with `meta={'playwright': True}`.

To use Scrapy Playwright, you must configure your `settings.py` file to include the `ScrapyPlaywrightDownloadHandler` and set the `TWISTED_REACTOR` for asyncio compatibility. Then, within your spider, create `scrapy.Request` objects with `meta={'playwright': True}`. You can also define page interactions using `playwright_page_methods` with `PageMethod` objects to wait for elements or perform actions before parsing.

import scrapy from scrapy_playwright.page import PageMethod # settings.py configuration (add these to your project's settings.py) # DOWNLOAD_HANDLERS = { # "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", # "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", # } # TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor" # PLAYWRIGHT_BROWSER_TYPE = "chromium" # or 'firefox', 'webkit' # PLAYWRIGHT_LAUNCH_OPTIONS = { # "headless": True, # Set to False for visual debugging # "timeout": 60000 # 60 seconds # } class MySpider(scrapy.Spider): name = "my_spider" start_urls = ["https://quotes.toscrape.com/js/"] def start_requests(self): for url in self.start_urls: yield scrapy.Request( url, meta={ "playwright": True, "playwright_page_methods": [ PageMethod("wait_for_selector", "div.quote"), # PageMethod("screenshot", path="screenshot.png", full_page=True), ], }, callback=self.parse_quotes, ) async def parse_quotes(self, response): # The response is now a PlaywrightResponse object with rendered content for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('small.author::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), } next_page = response.css('li.next a::attr(href)').get() if next_page is not None: yield scrapy.Request(response.urljoin(next_page), meta={'playwright': True, 'playwright_page_methods': [PageMethod("wait_for_selector", "div.quote")]}, callback=self.parse_quotes)

scrapy --version

Debug

Known issues

breakingPython 3.8 support was dropped in `scrapy-playwright==0.0.44`. Users on Python 3.8 or older must upgrade their Python version to >=3.10 to use recent versions.

fix

Upgrade Python to 3.10 or newer. Ensure your `requires_python` is set correctly.

affects: >=0.0.44

breakingThe import path for `HTTP11DownloadHandler` was updated in `scrapy-playwright==0.0.45`. This might affect projects with highly customized download handler setups or direct imports.

fix

Review and update any custom configurations or direct imports related to Scrapy's HTTP download handlers if they are dependent on `scrapy-playwright`'s internal path for this component.

affects: >=0.0.45

deprecatedPositional argument handling for the function passed to the `PLAYWRIGHT_PROCESS_REQUEST_HEADERS` setting was deprecated in version 0.0.41. Arguments should now be handled by keyword.

fix

Update custom `PLAYWRIGHT_PROCESS_REQUEST_HEADERS` functions to accept keyword arguments instead of positional ones.

affects: >=0.0.41

gotchaPlaywright requires browser binaries to be installed separately after the Python package. The `pip install playwright` command does not install these binaries by default. Without them, Playwright will not function.

fix

Run `playwright install` in your terminal after installing the Python package to download the necessary browser binaries (Chromium, Firefox, WebKit).

affects: All

gotchaScrapy must be configured to use an asyncio-compatible Twisted reactor (e.g., `twisted.internet.asyncioreactor.AsyncioSelectorReactor`) for `scrapy-playwright` to work correctly. Failing to do so will lead to asynchronous request failures.

fix

Add `TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'` to your `settings.py` file.

affects: All

gotchaPlaywright instances consume significant memory and CPU. Running many concurrent Playwright pages can exhaust system resources, leading to crashes or slow performance. This is especially critical when running non-headless browsers.

fix

Manage concurrency using Scrapy's `CONCURRENT_REQUESTS` and `CONCURRENT_REQUESTS_PER_DOMAIN` settings. Additionally, `PLAYWRIGHT_MAX_PAGES_PER_CONTEXT` can limit Playwright's parallel page usage. Always use `headless=True` in `PLAYWRIGHT_LAUNCH_OPTIONS` unless debugging visually.

affects: All

gotchaPlaywright's `Page.route` and `Page.unroute` methods are used internally by `scrapy-playwright`. Directly using these methods in user code can interfere with the library's functionality and lead to unexpected behavior.

fix

Avoid directly calling `Page.route` or `Page.unroute` on Playwright page objects obtained via `response.meta['playwright_page']` unless you have a deep understanding of the internal workings and are prepared for potential conflicts.

affects: All

Upgrade

Version history

0.0.47latest on PyPI · released Jun 13, 2026

Audit

Dependencies

ScrapyrequiredCore web scraping framework; requires >=2.7

playwrightrequiredHeadless browser automation library; requires >=1.40 (Python version)

Agent activity

26 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/scrapy-plugins/scrapy-playwright ↗

packagehttps://pypi.org/project/scrapy-playwright/ ↗