Registry / ai-ml / embedding-reader

embedding-reader

library1.8.1pypypi✓ verified 86d ago

embedding-reader is a Python library designed for efficiently reading embeddings from various file formats, including HDF5, Parquet, JSON, TSV, and NumPy with memory mapping (mmap). It also provides functionality to download embeddings from Hugging Face datasets. The library focuses on performance and ease of use for large-scale embedding datasets. The current version is 1.8.1, and it maintains a regular release cadence, often with monthly or bi-monthly updates.

ai-ml data

pip install embedding-reader

github ↗package ↗

INSTALL

IMPORT

SIG · EMBEDDING-READER

embedding-reader

ai-mlpythonv1.8.1

Install

10.1s avg

Import

1549ms

Disk

333MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.8.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 1.611s · 341.8MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 10.1s · import 1.488s · 310MB

333MB installed

● package 333MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

EmbeddingReader

✓ from embedding_reader import EmbeddingReader

✗ from embedding_reader.embedding_reader import EmbeddingReader

The main class is directly importable from the top-level package.

This quickstart demonstrates how to initialize the `EmbeddingReader` and read embeddings and their corresponding IDs from a file. It includes a setup for a dummy HDF5 file to make the example runnable. For actual use, replace `dummy_h5_path` with your local file path or a Hugging Face dataset identifier (e.g., `hf://facebook/dolly-v2-7b-embeddings`). Remember to specify the `embedding_column_name` and `id_column_name` if they differ from the defaults (`embedding` and `id` respectively).

import numpy as np import os from embedding_reader import EmbeddingReader # Create a dummy HDF5 file for demonstration # In a real scenario, you would point to an existing file or HF dataset dummy_h5_path = "./dummy_embeddings.h5" if not os.path.exists(dummy_h5_path): import h5py with h5py.File(dummy_h5_path, 'w') as f: f.create_dataset('embeddings', data=np.random.rand(10, 5)) f.create_dataset('ids', data=np.arange(10).astype(str)) # Initialize the EmbeddingReader # For Hugging Face datasets, use path='hf://organization/dataset_name' reader = EmbeddingReader( path=dummy_h5_path, embedding_column_name='embeddings', id_column_name='ids' ) # Read embeddings and IDs embeddings = reader.read_embeddings() ids = reader.read_ids() print(f"Read {len(embeddings)} embeddings of shape {embeddings.shape[1]}") print(f"First 3 IDs: {ids[:3]}") # Clean up dummy file os.remove(dummy_h5_path)

Debug

Known issues

gotchaWhen reading from Hugging Face, ensure you use the `hf://` prefix (e.g., `hf://organisation/dataset_name`) and provide `dataset_name` and `subfolder` parameters correctly if the dataset is not at the root or has multiple configurations.

fix

Always prepend `hf://` for Hugging Face paths. For multi-config datasets, specify `dataset_name` and `subfolder` during `EmbeddingReader` initialization or ensure `path` points to a specific config.

affects: All versions

breakingIn versions 1.7.0+, new `dataset_name` and `subfolder` parameters were introduced for better Hugging Face integration. Code relying on implicit path resolution for HF datasets might need adjustment.

fix

Explicitly pass `dataset_name` and `subfolder` parameters to the `EmbeddingReader` constructor when dealing with Hugging Face datasets, especially those with multiple configurations or nested structures.

affects: >=1.7.0

gotchaThe library infers the file format from the file extension. If your file has a non-standard extension or no extension, you might need to explicitly specify the `file_format` parameter to `EmbeddingReader`.

fix

If automatic format detection fails, provide `file_format='hdf5'`, `file_format='parquet'`, etc., in the `EmbeddingReader` constructor.

affects: All versions

gotchaFor very large NumPy (`.npy`) embedding files, enabling `mmap_mode='r'` in the `EmbeddingReader` constructor can significantly reduce memory usage by memory-mapping the file instead of loading it entirely into RAM.

fix

When dealing with large .npy files, initialize `EmbeddingReader` with `mmap_mode='r'` (e.g., `EmbeddingReader(path='file.npy', mmap_mode='r')`).

affects: >=1.6.0

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'embedding_reader'

The `embedding-reader` package is either not installed or installed in a different environment.

fix

Ensure the package is installed in your current Python environment using `pip install embedding-reader`.

FileNotFoundError: [Errno 2] No such file or directory: 'your/path/to/embeddings.h5'

The path provided to `EmbeddingReader` does not point to an existing file or directory.

fix

Double-check the file path. Ensure it's absolute or relative to your current working directory. If it's a remote path, confirm network access and correct URI (e.g., `s3://`, `hf://`).

ValueError: Unknown format for file: 'embeddings.dat'

The library could not infer the file format from the extension, or the extension is not supported.

fix

Rename the file with a supported extension (e.g., `.h5`, `.parquet`, `.npy`) or explicitly specify the `file_format` parameter during `EmbeddingReader` initialization (e.g., `EmbeddingReader(path='embeddings.dat', file_format='hdf5')`).

KeyError: 'embeddings' (when reading HDF5/Parquet)

The specified `embedding_column_name` or `id_column_name` (defaults are 'embedding' and 'id') does not exist in the source file.

fix

Inspect your HDF5/Parquet file to confirm the actual column names. Then, pass the correct names to the `EmbeddingReader` constructor: `EmbeddingReader(..., embedding_column_name='my_embeddings_key', id_column_name='doc_id')`.

Upgrade

Version history

1.8.1latest on PyPI · released Aug 9, 2025

Audit

Dependencies

numpyrequiredCore for numerical operations and array handling.

h5pyrequiredRequired for reading HDF5 format embedding files.

pandasrequiredUsed for reading Parquet, JSON, and TSV formats, particularly for structured data.

pyarrowrequiredUnderpins Parquet file reading, often used with Pandas.

fsspecrequiredAbstracts away filesystem details, enabling reading from local, S3, GCS, HF Hub, etc.

tqdmrequiredProvides progress bars for operations, especially during large file reads or downloads.

Agent activity

21 hits · last 30 days

node