Registry / data / fastcdc

fastcdc

JSON →
library1.7.0pypypi✓ verified 81d ago

FastCDC is a pure Python implementation of the Fast Content-Defined Chunking (CDC) algorithm, providing variable-size chunking based on content similarity. Current version is 1.7.0, supporting Python 3.7+.

pip install fastcdc
INSTALL
IMPORT
SIG · FASTCDC
F
fastcdc
datapythonv1.7.0
Install
2.5s avg
Import
Disk
20MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v1.7.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.95 runs
installs and imports cleanly · install 0.0s · import 0.000s · 21.1MB
glibc
py 3.103.95 runs
installs and imports cleanly · install 2.5s · import 0.000s · 23MB
20MB installed
● package 20MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

FastCDC
from fastcdc import fastcdc
from fastcdc import FastCDC

Demonstrates basic usage: instantiate FastCDC with average chunk size, then chunk bytes.

from fastcdc import FastCDC # Create a FastCDC instance with desired average chunk size cdc = FastCDC(avg_size=4096) # 4KB average chunk size # Chunk a bytes object data = b"""Some large binary data repeated many times to demonstrate chunking.""" * 100 chunks = list(cdc.chunk(data)) print(f"Number of chunks: {len(chunks)}") for offset, length, chunk_hash in chunks[:3]: print(f"Offset: {offset}, Length: {length}, Hash: {chunk_hash.hex()}")
Debug
Known issues
gotchaThe chunk() method returns an iterator of tuples (offset, length, hash). The chunk data must be fetched separately from the original bytes via data[offset:offset+length].
fix
Use data[offset:offset+length] to get chunk bytes after receiving offset and length.
affects: all
gotchaThe hash object is a bytes object of length 8 (64-bit) by default. This may not be suitable for deduplication at scale; consider using a custom hash function or additional hashing.
fix
For stronger hashing, pass a custom hash function to FastCDC or re-hash the chunk bytes with SHA-256.
affects: all
Upgrade
Version history
1.7.0latest on PyPI · released Jun 27, 2024
Audit
Dependencies

No dependency data recorded yet.

Agent activity
4 hits · last 30 days
node
4
Resources
fastcdc — pip install fastcdc · libregistry