dvc-http

library2.32.0pypypi✓ verified 24d ago

dvc-http is a plugin for Data Version Control (DVC) that provides support for HTTP and HTTPS remotes, allowing DVC to store and retrieve data from web servers. It leverages the `fsspec` library to provide filesystem-like access over HTTP(S). The current version is 2.32.0, and the project maintains an active release cadence with frequent updates.

data http-networking devops

pip install dvc-http

github ↗package ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · DVC-HTTP

dvc-http

datapythonv2.32.0

Install

4.2s avg

Import

289ms

Disk

28MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.32.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.312s · 28.8MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 4.2s · import 0.266s · 31MB

28MB installed

● package 28MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

fsspec.filesystem('http')

✓ import fsspec fs = fsspec.filesystem('http')

dvc-http registers itself with fsspec to handle 'http' and 'https' protocols. Direct imports from `dvc_http` are typically not needed for end-user functionality, as interaction is primarily through DVC's CLI or fsspec's API.

This quickstart demonstrates how to use `dvc-http` indirectly via the `fsspec` interface to read a file from an HTTP URL. When `dvc-http` is installed, `fsspec.filesystem('http')` will automatically use its backend to handle HTTP operations. For DVC CLI usage, simply configure an HTTP remote (e.g., `dvc remote add myremote http://example.com/data`).

import fsspec import os # dvc-http registers itself with fsspec to handle 'http' and 'https' protocols. # No direct import from dvc_http is typically needed for basic usage. try: # This will use dvc-http's implementation if installed fs = fsspec.filesystem("http") # Using a placeholder public URL for demonstration file_url = "http://www.textfiles.com/100/abacus.txt" print(f"Attempting to read from: {file_url}") with fs.open(file_url, "r", encoding="utf-8") as f: content = f.read(100) # Read first 100 characters print(f"\nSuccessfully read from {file_url}:") print("--- Content Snippet ---") print(content) print("-----------------------") except Exception as e: print(f"\nAn error occurred: {e}") print("Ensure dvc-http is installed and the URL is accessible.")

dvc --version

Debug

Known issues

breakingTimeout parameters for HTTP operations have changed. `sock_read_timeout` and `sock_connect_timeout` are no longer supported.

fix

Use `read_timeout` and `connect_timeout` instead. For example, pass `connect_timeout=10, read_timeout=30` to your DVC configuration or fsspec calls.

affects: >=2.29.0

gotchadvc-http no longer has `dvc` as a direct runtime dependency, nor does it import from `dvc` internally.

fix

This change primarily affects library internals and dependency resolution. Users typically install `dvc` and `dvc-http` separately. If you were relying on `dvc-http` implicitly pulling in `dvc` or on internal `dvc` imports from `dvc-http`, adjust your environment and code accordingly. For most users, this means `dvc-http` is more independent and modular.

affects: >=2.30.2

gotchaWhile `dvc-http`'s `fsspec[http]` dependency is handled automatically by pip, ensure `dvc-http` (or `fsspec[http]`) is correctly installed if you encounter 'protocol not found' errors when trying to use `fsspec.filesystem('http')`.

fix

Verify installation with `pip show dvc-http` and `pip show fsspec`. Reinstall if necessary using `pip install dvc-http`.

affects: all

Errors

Common errors & fixes

Timeout on reading data from socket

This error often occurs when pushing large files to an HTTP remote, indicating that the underlying `aiohttp` library's socket read timeout was exceeded before the entire file could be uploaded.

fix

Increase the `http.timeout` setting for your DVC remote using `dvc remote modify --local myremote http.timeout <seconds>` or adjust server-side timeouts if possible. A value of 0 disables the timeout for read operations.

ERROR: unexpected error - 401 Client Error: Unauthorized for url: http://...

DVC failed to authenticate with the HTTP or HTTPS remote server, usually due to incorrect username, password, or an improperly configured authentication method (`auth`, `custom_auth_header`).

fix

Verify and correctly configure your authentication details using `dvc remote modify --local myremote auth <method>` (e.g., `basic`, `custom`) along with `user`, `password`, or `custom_auth_header` as required by your remote. Ensure sensitive values are stored locally or via environment variables.

ImportError: cannot import name 'fsspec_loop' from 'fsspec.asyn'` or `ModuleNotFoundError: No module named 'public'

These are typically dependency conflicts where the installed `fsspec` version (or another underlying dependency like `flufl.lock`) is incompatible with the `dvc` or `dvc-http` version, often because `fsspec_loop` was removed in newer `fsspec` versions or the `public` module is missing.

fix

Downgrade `fsspec` to a version compatible with your DVC installation (e.g., `pip install fsspec==2022.11.0`). Ensure all DVC and `dvc-http` dependencies are correctly installed and compatible with your Python environment, possibly by reinstalling DVC with its HTTP extra: `pip install 'dvc[http]' --force-reinstall`.

[Errno 2] No such file or directory: '/tmp/tmp...dvc-cache/files/md5/...'` or `FileNotFoundError: [Errno 2] No such file or directory

DVC cannot find a referenced data file either in its local cache or on the configured HTTP remote. This can happen if the data was not successfully pushed to the remote, the local cache is corrupted, or the remote URL/path is incorrect.

fix

First, ensure the data was successfully pushed to the remote using `dvc push`. If the issue persists, try clearing and relinking the local cache with `dvc cache dir --unset` followed by `dvc checkout --relink`, or explicitly specifying the cache directory. Verify that the configured remote URL is accessible and correct.

Upgrade

Version history

2.32.0latest on PyPI · released Dec 13, 2023

Audit

Dependencies

fsspec[http]requiredCore dependency providing the filesystem interface for HTTP/HTTPS. This is automatically installed with `dvc-http`.

Agent activity

16 hits · last 30 days

node

OpenAI (training)

Amazon

Resources

githubhttps://github.com/iterative/dvc-http ↗

packagehttps://pypi.org/project/dvc-http/ ↗

homepagehttp://dvc.org ↗

docshttps://dvc.org/doc ↗