dvc-data

library3.18.3pypypi✓ verified 25d ago

dvc-data is DVC's core data management subsystem, providing functionalities for hashing, indexing, caching, and interacting with various storage backends. As a foundational library for DVC, it receives frequent patch and minor releases to enhance performance, expand compatibility, and fix bugs.

data devops

pip install dvc-data

github ↗package ↗

INSTALL

IMPORT

SIG · DVC-DATA

dvc-data

datapythonv3.18.3

Install

3.3s avg

Import

569ms

Disk

35MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v3.18.3 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.915 runs

installs and imports cleanly · install 0.0s · import 0.599s · 36.6MB

glibc

py 3.10–3.915 runs

installs and imports cleanly · install 3.3s · import 0.539s · 37MB

35MB installed

● package 35MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

DataIndex

✓ from dvc_data.index import DataIndex

Used for managing collections of data entries, often representing directories.

HashFile

✓ from dvc_data.hashfile import HashFile

Represents a single file with content-addressable hashing information.

HashFileDB

✓ from dvc_data.hashfile import HashFileDB

Manages a database of HashFile objects, similar to Git's object store.

This quickstart demonstrates the core functionality of `dvc-data` by creating a `DataIndex` and a `DataIndexEntry` for a local file, simulating how DVC tracks data. It uses `dvc_data.hashfile.hash.file_md5` to calculate the MD5 hash and stores it within the index entry.

import os from dvc_data.index import DataIndex, DataIndexEntry from dvc_data.hashfile.hash import file_md5 # Create a dummy file with open('data.txt', 'w') as f: f.write('hello dvc-data world') # Calculate its MD5 hash md5_hash = file_md5('data.txt', os.fspath, None) # Create a DataIndexEntry for the file entry = DataIndexEntry( path='data.txt', hash_info={'name': 'md5', 'value': md5_hash} ) # Add the entry to a DataIndex index = DataIndex() index[('data.txt',)] = entry print(f"Created DataIndex with entry for 'data.txt': {index.has_entries}") print(f"MD5 for 'data.txt': {index[('data.txt',)].hash_info['value']}") # Clean up os.remove('data.txt')

Debug

Known issues

breakingDVC 3.0 introduced significant changes to file hashing (removed CRLF conversion) and cache storage locations. Old '.dvc' files containing deprecated keys (like 'metric', 'param', 'plot', 'cmd') will cause validation errors and older cache structures are incompatible.

fix

Run `dvc cache migrate` to move existing DVC 2.x cache data to the 3.x format. Manually remove deprecated keys from `.dvc` files or regenerate them using `dvc add` if issues persist. Ensure consistent line endings in text files across platforms.

affects: >=3.0.0

gotchaThe `bulk_exists` method in earlier 3.x versions had issues with filesystems not implementing `ls` or when handling duplicate hashes, potentially leading to incorrect results during data existence checks.

fix

Upgrade to `dvc-data` version 3.18.3 or newer to benefit from fixes for `bulk_exists` behavior across various filesystems and scenarios.

affects: <3.18.3

gotchaEarlier versions might mishandle protocol information when filtering changed metadata in `_filter_changed`, potentially leading to inaccurate detection of data changes, especially with custom protocols.

fix

Upgrade to `dvc-data` version 3.18.3 or newer to ensure correct `protocol` propagation in metadata operations.

affects: <3.18.3

gotchaWhen working with DVC commands like `dvc pull`, `fetch`, or `push` (which rely on `dvc-data`), users might encounter '[Errno 24] Too many open files' errors, especially on macOS with S3 remotes, when using many `--jobs`.

fix

Increase the operating system's open file descriptor limit (`ulimit -n` on UNIX-like systems) or use a lower value for the `--jobs` parameter.

affects: All versions

Errors

Common errors & fixes

WARNING: Cache 'xxxx' not found. or ERROR: failed to pull data from the cloud

This error occurs when DVC cannot find the expected data files in its local cache or the configured remote storage, often because the data was committed to Git (via .dvc files) but not pushed to the DVC remote, or the cache is corrupted.

fix

Ensure all DVC-tracked data is pushed to the remote by running `dvc push`. If the local cache is suspected to be incomplete or corrupted, run `dvc pull` to re-synchronize from the remote. For S3 remotes, you can enable checksum verification with `dvc remote modify <remote_name> verify true` to detect data integrity issues.

ERROR: unexpected error - [Errno 2] No such file or directory: 'path/to/file'

This generic file system error frequently arises in DVC operations (like `dvc pull`, `dvc import`, `dvc repro`) when a file or directory referenced in a .dvc file or by DVC's internal mechanisms cannot be found. Common reasons include incorrect paths in .dvc files, misconfigured cache directories (`dvc config cache.dir`), issues with symbolic or hard links, or operating system limitations such as overly long file paths on Windows.

fix

Verify that all file paths specified in your .dvc files and DVC configuration are correct and accessible. Check your `dvc config cache.dir` setting. If using symbolic links or hard links, ensure your file system supports them; otherwise, configure `dvc config cache.type copy`. On Windows, enable long path support if dealing with deep directory structures to prevent path limit issues.

ModuleNotFoundError: No module named 'public' OR ModuleNotFoundError: No module named 'dvc' OR ModuleNotFoundError: No module named 'src'

These Python `ModuleNotFoundError` issues indicate that DVC or one of its essential dependencies (like `dvc-data` or `public` from `flufl.lock`) is not correctly installed or the Python environment where DVC was installed is not active.

fix

Activate your Python virtual environment (if applicable). Reinstall DVC and ensure all necessary dependencies are installed by running `pip install dvc --upgrade` or `pip install dvc[<remote_type>] --upgrade` for specific remote storage support. If 'public' is specifically missing, try `pip install public`.

ERROR: failed to import 'data' from 'git@host:repo.git'. - Git failed to fetch ref from... OR dvc.exceptions.OutputNotFoundError: Unable to find DVC file with output 'Dogs.dvc'.

These errors occur during `dvc import` or `dvc get` operations, indicating issues with accessing or locating data from a source repository. This could be due to an incorrect repository URL or path to the DVC-tracked file, failed Git authentication (e.g., missing SSH keys), or if the target data is not correctly tracked by DVC in the source repository (e.g., it's an external output or not a .dvc file).

fix

Double-check the Git repository URL and the exact path to the DVC-tracked data (.dvc file or dvc.yaml output) within that repository. Ensure your Git client is correctly configured with necessary credentials (e.e.g., SSH keys or token) to access the remote repository. Confirm that the source DVC repository has a `dvc remote default` configured for its cache, and that the data is tracked directly (not as an external output via `--external`).

Upgrade

Version history

3.18.3latest on PyPI · released Feb 27, 2026

Audit

Dependencies

dvc-objectsrequiredProvides core object storage abstractions; dvc-data builds upon it.

fsspecrequiredAbstracts away filesystem operations, used for various storage backends.

boto3optionalRequired for S3 remote storage (installed via `dvc-data[s3]` or `dvc-data[all]`).

azure-storage-bloboptionalRequired for Azure remote storage (installed via `dvc-data[azure]` or `dvc-data[all]`).

gcsfsoptionalRequired for Google Cloud Storage (installed via `dvc-data[gs]` or `dvc-data[all]`).

Agent activity

10 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/iterative/dvc-data ↗

packagehttps://pypi.org/project/dvc-data/ ↗