Rechunker is a Python package designed for efficient and scalable manipulation of the chunk structure of chunked array formats, such as Zarr and TileDB. It takes an input array (or group of arrays) from persistent storage and writes out a new array with the same data but a different chunking scheme, often utilizing an intermediate temporary store. It is currently at version 0.5.4 and is actively maintained by the Pangeo community, with regular releases addressing compatibility and bug fixes.
pip install rechunkerVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to rechunk a Zarr array from an initial chunking scheme to a new one. It defines source, intermediate, and target Zarr stores, specifies the target chunk size and maximum memory per worker, creates a rechunking plan, and then executes it.
Always use the latest version of `rechunker` to ensure compatibility with recent `xarray` and `dask` versions, or refer to release notes for specific version requirements.
Understand that `rechunker` is designed for disk-to-disk rechunking and explicitly manages intermediate storage to avoid out-of-memory errors on large datasets that Dask's in-memory `rechunk` might encounter. Allocate sufficient `max_mem` and `temp_store`.
Ensure input arrays have uniform chunk sizes before passing them to `rechunker`. If using Dask arrays with non-uniform chunks, consider explicitly rechunking them to a uniform size using `dask.array.rechunk` beforehand, or manually ensure uniformity.
Update `rechunker`, `xarray`, and `zarr` to their latest compatible versions. Verify if `consolidated=True` is used when writing to Zarr if applicable, and ensure all attributes are properly copied (addressed in `v0.5.1`).
After creating the `rechunked` plan, call its `.execute()` method to perform the rechunking operation: `rechunked_plan = rechunk(...); result_array = rechunked_plan.execute()`.
Review `target_chunks` and `max_mem` parameters to ensure they are valid and sensible for the input array's dimensions and data type. Check GitHub issues for similar reported problems and potential workarounds, especially for edge cases with very small or very large dimensions.