Registry / data / pyspark-huggingface

pyspark-huggingface

library2.1.0pypypiunverified

pyspark-huggingface is a Spark Data Source for seamlessly accessing 🤗 Hugging Face Datasets as Spark DataFrames. It enables streaming datasets from the Hub, applying projection and predicate filters, and saving Spark DataFrames back to Hugging Face as Parquet files with fast, deduplicated uploads. It supports authentication via `huggingface-cli login` or tokens, and is compatible with Spark 4 (with auto-import) as well as backporting functionality for Spark 3.5, 3.4, and 3.3. The current version is 2.1.0 and it is actively maintained.

data ai-ml database

pip install pyspark-huggingface

package ↗

INSTALL

IMPORT

SIG · PYSPARK-HUGGINGFAC

pyspark-huggingface

datapythonv2.1.0

Install

16.4s avg

Import

—

Disk

388MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.1.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

installs and imports cleanly · install 0.0s · import 0.000s · 398MB

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 16.4s · import 0.000s · 368MB

388MB installed

● package 388MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

pyspark_huggingface

✓ import pyspark_huggingface

✗ import pyspark_huggingface

This quickstart initializes a Spark session and demonstrates how to read a public Hugging Face dataset (stanfordnlp/imdb) into a PySpark DataFrame. It also shows where to configure authentication for private datasets or write operations. For Spark 3.x users, an explicit `import pyspark_huggingface` might be necessary.

from pyspark.sql import SparkSession import os # Initialize Spark Session spark = SparkSession.builder \ .appName("HuggingFaceSpark") \ .getOrCreate() # For Spark 3.x, explicitly import to enable the data source: # import pyspark_huggingface # Read a public dataset from Hugging Face # Replace 'hf_token_xxxx' with your actual token if accessing private/gated datasets hf_token = os.environ.get('HF_TOKEN', '') # Use a real token for private datasets print("Reading stanfordnlp/imdb dataset...") df = spark.read \ .format("huggingface") \ .option("token", hf_token) \ .load("stanfordnlp/imdb") print("Schema:") df.printSchema() print("First 5 rows:") df.show(5, truncate=False) # Example of saving a DataFrame to Hugging Face (requires write token and dataset repo name) # try: # print("Saving a sample DataFrame to Hugging Face...") # sample_data = [("hello", "world"), ("spark", "huggingface")] # sample_df = spark.createDataFrame(sample_data, ["col1", "col2"]) # sample_df.write \ # .format("huggingface") \ # .option("token", hf_token) \ # .mode("overwrite") \ # .save("your_username/your_dataset_name") # print("DataFrame saved successfully.") # except Exception as e: # print(f"Could not save DataFrame to Hugging Face: {e}") spark.stop()

Debug

Known issues

gotchaWhen using `pyspark-huggingface` with PySpark 3.x (versions 3.3, 3.4, 3.5), you *must* explicitly `import pyspark_huggingface` in your code to enable the 'huggingface' data source format. This is not needed for PySpark 4+ as it's auto-imported.

fix

Add `import pyspark_huggingface` at the beginning of your Spark application when using PySpark 3.x.

affects: <=3.5

gotchaTo read private/gated Hugging Face datasets or to write Spark DataFrames to the Hugging Face Hub, you need to authenticate. Not providing a valid token will lead to access errors.

fix

Authenticate using `huggingface-cli login` in your environment or pass your Hugging Face token directly via the `.option("token", "hf_xxxx")` to the `spark.read` or `spark.write` calls, or set the `HF_TOKEN` environment variable.

affects: All

gotchaFor optimal performance, especially with large Parquet datasets, apply filters and select columns during the `spark.read.format('huggingface').option(...)` stage. This leverages Parquet metadata to skip unnecessary data, reducing I/O and processing.

fix

Use `.option("filters", '[("column_name", ">", value)]')` and `.option("columns", '["col1", "col2"]')` when loading the dataset. Example: `.option("filters", '[("language_score", ">", 0.99)]').option("columns", '["text", "language_score"]')`.

affects: All

gotchaDownloading large models or datasets can quickly exhaust disk space if the default Hugging Face cache directory (often in `/root/.cache/huggingface/hub`) is on a small root partition.

fix

Set the `HF_HUB_CACHE` environment variable to a path with sufficient available disk space *before* importing any `transformers` or `datasets` libraries. For example: `os.environ['HF_HUB_CACHE'] = "/path/to/large/storage"`.

affects: All

gotchaWhen uploading very large datasets with many shards to the Hugging Face Hub, you might encounter `HfHubHTTPError: 429 Client Error: Too Many Requests` due to hourly quotas.

fix

Ensure your `datasets` library (a common underlying dependency) is updated to version 2.15.0 or later, as this version includes improvements to handle such rate limits. `pip install --upgrade datasets`.

affects: <2.15.0 of `datasets` library

Upgrade

Version history

2.1.0latest on PyPI · released Apr 14, 2026

Audit

Dependencies

pysparkrequiredCore dependency for Apache Spark integration and DataFrame operations.

huggingface_hubrequiredRequired for authentication with the Hugging Face Hub and general Hub interactions.

pyarrowrequiredUsed under the hood for efficient reading and writing of Arrow/Parquet data formats.

Agent activity

16 hits · last 30 days

node

Amazon

OpenAI (training)

Resources

packagehttps://pypi.org/project/pyspark-huggingface/ ↗