High-level web crawling and scraping framework. Current version is 2.14.1 (Jan 2026). Requires Python >=3.10. Two major breaking changes in 2.13: start_requests() (sync) replaced by start() (async), and TWISTED_REACTOR now defaults to asyncio — both can silently break existing spiders.
Install & Compatibility
Where this runs
tested against v2.16.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
start_requests() (sync) was replaced by start() (async) in Scrapy 2.13. start_requests() still works but start() is the new preferred interface. Custom start_requests() overrides still function but cannot yield items directly.
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['https://example.com']
# New async start() method (2.13+) — preferred over start_requests()
async def start(self):
for url in self.start_urls:
yield scrapy.Request(url, callback=self.parse)
def parse(self, response):
yield {'title': response.css('title::text').get()}
Basic spider. Run with: scrapy crawl quotes -o output.json
import scrapy
class QuotesSpider(scrapy.Spider):
name = 'quotes'
start_urls = ['https://quotes.toscrape.com']
def parse(self, response):
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
# Follow pagination
next_page = response.css('li.next a::attr(href)').get()
if next_page:
yield response.follow(next_page, self.parse)
scrapy --version
Upgrade
Version history
Breaking-change detection hasn't run for this library yet.
Audit
Security & dependencies
CVE tracking and dependency tree are planned for a later release.