hyparquet

library1.25.6jsnpmunverified

Hyparquet is a pure JavaScript library for parsing Apache Parquet files directly in web browsers and Node.js environments. It specializes in efficient data retrieval from cloud storage by leveraging HTTP range requests, allowing for direct querying of Parquet files over the network without requiring a server-side intermediary. The library is dependency-free since 2023, offering a lightweight solution. Its current stable version is 1.25.6, with a release cadence that follows active development. Key differentiators include its ability to minimize data fetches through selective row and column filtering, comprehensive support for all Parquet types, encodings, and compression codecs, and inclusion of TypeScript definitions for improved developer experience. It is particularly well-suited for data engineering, data science, and machine learning applications where large datasets stored in Parquet format need to be accessed and processed client-side.

data database http-networking ai-ml

npm install hyparquet

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · HYPARQUET

hyparquet

datajavascriptv1.25.6

Install

—

Import

—

Disk

—

Pass rate

0/ 6

Env Coverage0 / 6

glibc

18–22

musl

18–22

Install & Compatibility

Where this runs

tested against v? · npm install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

node 18–226 runs

build_error

glibc

node 18–226 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

parquetReadObjects

✓ import { parquetReadObjects } from 'hyparquet'

✗ const { parquetReadObjects } = require('hyparquet')

Hyparquet is published as an ES module; CommonJS 'require()' is not supported directly. Use dynamic import() or ensure your Node.js environment supports ES modules.

asyncBufferFromUrl

✓ import { asyncBufferFromUrl } from 'hyparquet'

Used to create an AsyncBuffer from a URL, suitable for browser environments or Node.js network fetches. For local files in Node.js, use asyncBufferFromFile.

parquetMetadataAsync

✓ import { parquetMetadataAsync, parquetSchema } from 'hyparquet'

✗ import parquetMetadataAsync from 'hyparquet'

parquetMetadataAsync and parquetSchema are named exports, not default. They are crucial for inspecting file structure and statistics without full data loading.

AsyncBuffer

✓ import type { AsyncBuffer } from 'hyparquet'

This is a TypeScript type import for custom implementations of the AsyncBuffer interface.

Demonstrates how to fetch and parse a remote Parquet file in a browser or Node.js using HTTP range requests, filtering for specific columns and rows.

import { asyncBufferFromUrl, parquetReadObjects } from 'hyparquet' async function fetchData() { const url = 'https://hyperparam-public.s3.amazonaws.com/bunnies.parquet' // Wrap the URL for asynchronous fetching with HTTP range requests const file = await asyncBufferFromUrl({ url }) // Read objects, filtering by specific columns and rows for efficiency const data = await parquetReadObjects({ file, columns: ['Breed Name', 'Lifespan'], rowStart: 10, rowEnd: 20, }) console.log('Fetched data:', data) } fetchData().catch(console.error)

Debug

Known issues

breakingHyparquet is distributed exclusively as an ES module (ESM). Direct 'require()' calls for CommonJS environments are not supported.

fix

Use ESM import statements (e.g., `import { ... } from 'hyparquet'`) or dynamic `import()` for compatibility with older Node.js versions. Configure your project to handle ES modules.

affects: >=1.0.0

gotchaWhen reading remote Parquet files with `asyncBufferFromUrl`, efficient performance relies on the server supporting HTTP range requests. Without proper server support, the entire file might be downloaded.

fix

Ensure your HTTP server (e.g., S3, Google Cloud Storage) is configured to handle `Range` headers for partial content requests. Test with small files first.

affects: >=1.0.0

gotchaThe `num_rows` property from Parquet metadata is returned as a `BigInt`. Direct arithmetic operations or comparisons with standard `Number` types may lead to errors or unexpected results.

fix

Convert `BigInt` values to `Number` using `Number(metadata.num_rows)` when working with smaller numbers, or ensure your application logic correctly handles `BigInt` types for large row counts.

affects: >=1.0.0

gotchaTo optimize data fetching for large remote files, always specify `columns`, `rowStart`, and `rowEnd` parameters in `parquetReadObjects`. Failing to do so will result in downloading and parsing the entire file.

fix

Define `columns` to specify only the needed columns and `rowStart`/`rowEnd` to limit the row range, reducing network bandwidth and parsing overhead. Example: `parquetReadObjects({ file, columns: ['col1'], rowStart: 0, rowEnd: 100 })`.

affects: >=1.0.0

Errors

Common errors & fixes

TypeError: require is not a function

Attempting to import hyparquet using CommonJS `require()` syntax in a Node.js environment or bundler that expects ES modules.

fix

Change `const { ... } = require('hyparquet')` to `import { ... } from 'hyparquet'`. Ensure your Node.js project or bundler is configured for ES modules (e.g., 'type': 'module' in package.json or using .mjs extension).

TypeError: Cannot read properties of undefined (reading 'slice')

The `file` argument passed to `parquetReadObjects` or `parquetMetadataAsync` is not a valid `AsyncBuffer` instance or a compatible object.

fix

Ensure `file` is created using `asyncBufferFromUrl({ url })` for remote files or `asyncBufferFromFile('path/to/file.parquet')` for local Node.js files, or provide a custom object implementing the `AsyncBuffer` interface correctly.

Uncaught (in promise) TypeError: x.map is not a function

This error often occurs when attempting to map over a non-array result, sometimes indicating that the Parquet file was malformed, or the schema parsing returned an unexpected structure.

fix

Verify the integrity and structure of your Parquet file. Ensure the `parquetSchema` function is used correctly, and its output (e.g., `schema.children`) is what you expect to iterate over. Inspect `metadata` and `schema` objects before mapping.

Upgrade

Version history

1.25.6latest on npm

Audit

Dependencies

No dependency data recorded yet.

Agent activity

26 hits · last 30 days

node

Resources

githubhttps://github.com/hyparam/hyparquet ↗

packagehttps://www.npmjs.com/package/hyparquet ↗

homepagehttps://hyperparam.app ↗