HuggingFace library for loading, processing, and sharing datasets for ML. Provides load_dataset() for one-line access to 100k+ public datasets on the Hub, plus local file loading (CSV, JSON, Parquet, Arrow, audio, image, etc.). Built on Apache Arrow for memory-efficient, zero-copy data access. Package name on PyPI is 'datasets' (not 'huggingface-datasets'). Import name is also 'datasets'. CRITICAL: datasets 4.0 (July 2025) removed dataset loading scripts and trust_remote_code entirely. Many older community datasets relying on .py loading scripts now fail with datasets>=4.
pip install datasetsVerified import paths — ran on the pinned version, not inferred.
load_dataset() downloads and caches to HF_HOME. Streaming mode avoids full download for large datasets. map() with batched=True is significantly faster for large datasets. set_format() enables framework-specific tensor output without copying data.
Either: (1) pin datasets<4 as a workaround, or (2) find a Parquet-backed version of the dataset on the Hub, or (3) ask the dataset author to migrate to standard Parquet format. Passing trust_remote_code=True no longer silences the error — it raises a separate error.
Remove trust_remote_code from all load_dataset() calls when using datasets>=4.
Always upgrade together: pip install -U datasets pyarrow. Check compatibility in the datasets changelog for your target version.
Use named functions instead of lambdas for map(). Avoid referencing non-serializable objects inside map functions. Use batched=True for large datasets to avoid per-example overhead.
Pass download_mode='force_redownload' to bypass cache. Or delete the cached dataset from ~/.cache/huggingface/datasets/.
Set HF_TOKEN env var and accept the dataset license on huggingface.co before calling load_dataset().
pip install datasets. from datasets import load_dataset.
Ensure C/C++ build tools and Python development headers are installed in your environment before installing `datasets` with extras. For Alpine Linux, run `apk add build-base python3-dev`. For Debian/Ubuntu-based images, run `apt-get update && apt-get install -y build-essential python3-dev`.
Downgrade `datasets` to a version older than 4.0.0 (`pip install 'datasets<4.0.0'`) or ask the dataset author to convert the dataset to a standard format like Parquet.
Double-check the dataset name/path, ensure local files exist at the specified location, log in to Hugging Face (`huggingface-cli login`) if accessing private datasets, or clear the `datasets` cache and try again. For local files, explicitly specify the format (e.g., `load_dataset('csv', data_files='my_data.csv')`).For casting errors, explicitly define the `features` argument in `load_dataset` with the correct schema, or for CSV/JSON, manually provide `column_names` to `load_dataset`. For 'Invalid pattern', correct the glob pattern or use simpler file paths.
Install the library using `pip install datasets`. If already installed, check for shadowing files in your project directory and rename them. Ensure your virtual environment is activated if applicable.
Remove or update the problematic import statement (e.g., `from datasets.tasks import TextClassification`). The `datasets.tasks` module is generally not meant for direct user import in recent versions; task-related information is usually handled differently within the library's features. Upgrade `datasets` and `huggingface_hub` to their latest versions.