Registry / data / kerchunk

kerchunk

JSON →
library0.2.10pypypi✓ verified 84d ago

Kerchunk is a Python library for creating and manipulating chunked reference descriptions for cloud-optimized data access. It enables efficient reading of scientific data (e.g., NetCDF/HDF5) from remote storage without downloading entire files. The current version is 0.2.10, supporting Python >=3.11, with a stable but evolving API.

pip install kerchunk
INSTALL
IMPORT
SIG · KERCHUNK
K
kerchunk
datapythonv0.2.10
Install
5.1s avg
Import
1106ms
Disk
129MB
Pass rate
5/ 10
Env Coverage5 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v0.2.7 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.95 runs
build_error
glibc
py 3.103.95 runs
installs and imports cleanly · install 5.1s · import 1.106s · 125MB
129MB installed
● package 129MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

combine_kwargs
from kerchunk.combine import MultiZarrToZarr
from kerchunk import MultiZarrToZarr
MultiZarrToZarr is in the combine submodule, not top-level.
open_dataset
from kerchunk.hdf import KerchunkGroup
from kerchunk import open_dataset
For HDF5/NetCDF files, use KerchunkGroup or single_file_to_reference, no direct open_dataset.

Creates a single-file reference and opens it with xarray via Zarr engine.

import fsspec import xarray as xr from kerchunk.hdf import single_file_to_reference from kerchunk.combine import MultiZarrToZarr # Generate reference for a single file url = 's3://example-bucket/file.nc' # or local path fs = fsspec.filesystem('s3', anon=True) with fs.open(url) as f: h5chunks = single_file_to_reference(f, url) # Save reference as JSON import json with open('ref.json', 'w') as f: json.dump(h5chunks, f) # Open with xarray mapper = fsspec.get_mapper('reference://', fo='ref.json') ds = xr.open_dataset(mapper, engine='zarr') print(ds)
Debug
Known issues
breakingKerchunk 0.2.0 dropped Python 3.8 support. Use Python >=3.11 as of 0.2.10.
fix
Upgrade Python to 3.11 or later.
affects: >=0.2.0, !=0.2.10
gotchaFor multi-file concatenation, MultiZarrToZarr expects a list of reference dicts, not file paths. Passing file paths will raise cryptic errors.
fix
Generate single-file references first using single_file_to_reference, then pass to MultiZarrToZarr.
affects: all
deprecatedThe 'kerchunk.combine' module's 'concat' function is deprecated in favor of MultiZarrToZarr.
fix
Use MultiZarrToZarr with appropriate options.
affects: <0.2.0
Errors
Common errors & fixes
KeyError: 'refs'
Attempting to open a single-file reference dict directly with fsspec without using the correct mapper.
fix
Use fsspec.get_mapper('reference://', fo='ref.json') or wrap the dict in a ReferenceFileSystem. See quickstart.
AttributeError: module 'kerchunk' has no attribute 'combine'
Outdated kerchunk version (<0.2.0) or wrong import path. The combine module exists but is not imported by default from top-level.
fix
Use `from kerchunk.combine import MultiZarrToZarr` and ensure kerchunk>=0.2.0.
ValueError: unrecognized chunk manager: none
Opening a reference file with xarray without specifying engine='zarr' or using a non-Zarr engine.
fix
Always specify `engine='zarr'` when opening reference datasets in xarray.
Upgrade
Version history
0.2.10latest on PyPI · released Mar 30, 2026
Audit
Dependencies
fsspecrequiredRequired for file system abstractions and reference file system implementation.
xarrayoptionalOptional for integration with xarray for opening reference datasets.
Agent activity
31 hits · last 30 days
node
28
OpenAI (training)
1
Resources
kerchunk — pip install kerchunk · libregistry