parfive

library2.3.1pypypi✓ verified 85d ago

Parfive is an asynchronous HTTP and FTP parallel file downloader for Python. It leverages `asyncio` to efficiently download multiple files concurrently, providing features like progress bars, connection throttling, and retry mechanisms. The current version, 2.3.1, offers a robust asynchronous API for managing large-scale file transfers. Releases are made periodically to add features, address issues, and ensure compatibility with newer Python versions, maintaining an active development status.

http-networking data

pip install parfive

package ↗

INSTALL

IMPORT

SIG · PARFIVE

parfive

http-networkingpythonv2.3.1

Install

4.1s avg

Import

640ms

Disk

27MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.3.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 0.680s · 28MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 4.1s · import 0.600s · 30MB

27MB installed

● package 27MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Downloader

✓ from parfive import Downloader

✗ import parfive.Downloader

Downloader is a class directly exported by the parfive package, not a submodule.

This example demonstrates how to use `parfive.Downloader` to download multiple public files concurrently to a local directory. It shows how to initialize the downloader, add URLs, and await the results. The `asyncio.run()` function is used to execute the asynchronous main function.

import parfive import asyncio import os async def main(): # Define some public URLs to download urls = [ "https://raw.githubusercontent.com/sunpy/parfive/main/README.md", "https://raw.githubusercontent.com/sunpy/parfive/main/LICENSE" ] # Create a directory for downloads if it doesn't exist download_dir = "parfive_downloads" os.makedirs(download_dir, exist_ok=True) # Initialize the Downloader with a maximum of 5 concurrent connections # and display a progress bar. downloader = parfive.Downloader(max_conn=5, progress=True) # Add URLs to the downloader, specifying the local path for url in urls: downloader.add_url(url, path=download_dir) # Execute the downloads asynchronously print(f"Starting download of {len(urls)} files to '{download_dir}'...") results = await downloader.download() # Print the paths of the downloaded files print("Downloaded files:") for filepath in results: print(f"- {filepath}") # Optionally clean up the downloaded files # for filepath in results: # os.remove(filepath) # os.rmdir(download_dir) if __name__ == "__main__": asyncio.run(main())

parfive --version

Debug

Known issues

breakingVersion 2.0.0 introduced significant breaking changes, migrating the API to be fully `asyncio` native. The `Downloader` class signature and the return value of `download()` changed from a simple list of paths to a `parfive.results.Results` object which is a specialized list subclass.

fix

Review the official `parfive` documentation for the 2.x API. Ensure all `parfive` operations are `await`ed and run within an `asyncio` event loop. Adapt code to handle `Results` objects.

affects: >=2.0.0

gotchaAll `parfive` download operations are asynchronous and must be executed within an `asyncio` event loop. Forgetting to `await` the `downloader.download()` call or attempting to run it outside an event loop will lead to runtime errors.

fix

Ensure your code calls `downloader.download()` with `await` and that the execution context is an `asyncio` event loop (e.g., using `asyncio.run()` for top-level script execution, or within an `await`-able function in an existing event loop).

affects: All

gotchaBy default, `parfive` skips downloading a file if it already exists at the target path and has the correct size. While often desired, this can lead to stale files if the remote content changes without a size change. It also means you need to manage unique filenames or explicitly remove existing files if fresh downloads are always required.

fix

If fresh downloads are always needed, ensure target paths are unique (e.g., by including a timestamp) or explicitly delete existing files before calling `add_url`. Consider checking `overwrite=True` if that option exists or checking the `checksum`.

affects: All

gotchaThe `max_conn` (default 5) and `max_downloads` (default 10) parameters of `Downloader` control concurrency. Setting these too high can exhaust system resources (file descriptors, network sockets) or trigger rate limits on target servers, leading to slower downloads or connection errors.

fix

Tune `max_conn` and `max_downloads` based on your system's capabilities, network bandwidth, and the policies of the servers you are downloading from. Start with conservative values and increase gradually if performance allows.

affects: All

Errors

Common errors & fixes

RuntimeError: Event loop is already running

Attempting to call `asyncio.run()` in an environment where an event loop is already active (e.g., inside a Jupyter notebook or another async function).

fix

If in Jupyter, use `await` directly at the top level (if supported) or `nest_asyncio.apply()`. If within another async function, just `await` the parfive call without `asyncio.run()`.

TypeError: object asyncio.tasks.Task can't be awaited

Forgetting the `await` keyword before calling an asynchronous function or method, specifically `downloader.download()`.

fix

Ensure you use `await downloader.download()` to properly execute the asynchronous download process and retrieve its results.

AttributeError: module 'parfive' has no attribute 'Downloader'

Trying to access `parfive.Downloader` when `Downloader` is directly exported by the package and should be imported with `from parfive import Downloader`.

fix

Change `parfive.Downloader` to `Downloader` after using `from parfive import Downloader` or ensure you are importing the `Downloader` class correctly.

Upgrade

Version history

2.3.1latest on PyPI · released Jan 27, 2026

Audit

Dependencies

aiohttprequiredCore HTTP client for asynchronous web requests.

tqdmrequiredProvides progress bar functionality during downloads.

asyncio-throttlerequiredUsed for managing and limiting concurrent tasks.

Agent activity

6 hits · last 30 days

node

Resources

packagehttps://pypi.org/project/parfive/ ↗