Registry / ai-ml / megatron-energon

megatron-energon

library7.4.0pypypi✓ verified 84d ago

NVIDIA Megatron-Energon is a multi-modal data loader library for large-scale deep learning, particularly for training large language models (LLMs) and vision-language models. It supports tar-based WebDataset, JSONL files, and polylithic datasets, with features like caching, AV decoding, and FUSE mount. Current version is 7.3.2, with active development and a release cadence of roughly monthly.

ai-ml llm-agents data workflow

pip install megatron-energon

github ↗package ↗

INSTALL

IMPORT

SIG · MEGATRON-ENERGON

megatron-energon

ai-mlpythonv7.4.0

Install

—

Import

—

Disk

—

Pass rate

0/ 10

Env Coverage0 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v7.4.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✕ build_error

1/2 runs

py 3.11

✕ build_error

1/2 runs

py 3.12

✕ build_error

1/2 runs

py 3.13

✕ build_error

1/2 runs

py 3.9

✕ build_error

✕ timeout

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

get_train_dataset

✓ from megatron.energon import get_train_dataset

✗ from energon import get_train_dataset

Incorrect import path; module is under megatron.energon.

WorkerConfig

✓ from megatron.energon import WorkerConfig

✗ from energon import WorkerConfig

Incorrect import path; module is under megatron.energon.

Webdataset

✓ from megatron.energon import Webdataset

Correct import for Webdataset class.

Minimal example loading a training dataset with Megatron-Energon.

import os from megatron.energon import get_train_dataset, WorkerConfig # Create a simple dataset worker_config = WorkerConfig( rank=0, world_size=1, num_workers=2, ) dataset = get_train_dataset( path=os.environ.get('DATASET_PATH', 'path/to/dataset'), worker_config=worker_config, batch_size=4, shuffle_buffer_size=100, ) for batch in dataset: # Each batch is a dict with keys like 'rgb', 'json' print(batch.keys()) break

megatron-energon --version

Debug

Known issues

breakingVersion 7.0.0 introduced polylithic datasets and a new AVDecoder, breaking datasets created with v6.x. Existing datasets may need re-preparation.

fix

Run `energon prepare` again or use an older version (6.0.1) if you must keep legacy format.

affects: >=7.0.0

breakingIn version 6.0.0, the save/restore mechanism changed: worker dimension is now the outer dimension. Checkpoint compatibility broken.

fix

Implement new save/restore logic as per updated docs; old checkpoints are not compatible.

affects: >=6.0.0 <7.0.0

gotchaThe `disable_cache` option (for ITarReader) is only available from 7.3.2 onward. Before that, caching may cause thread-safety issues.

fix

Upgrade to >=7.3.2 or avoid concurrent tar readers.

affects: <7.3.2

deprecatedThe use of fsspec was replaced by EPath in 6.0.1. Code relying on fsspec paths may break.

fix

Update to EPath paths as described in the migration guide.

affects: <6.0.1

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'energon'

Attempting to import directly from 'energon' instead of 'megatron.energon'.

fix

Use `from megatron.energon import ...`

sqlite3.OperationalError: locking protocol

Concurrent SQLite access in cache database without proper retry logic.

fix

Upgrade to >=7.2.2 which includes a fix for this locking issue.

ValueError: Do not know how to load sample, given the available subflavors

The dataset metadata does not match the sample type you are requesting.

fix

Ensure your dataset's `.nvs` file includes the correct subflavors (e.g., 'image', 'video', 'audio').

Upgrade

Version history

7.4.0latest on PyPI · released Jun 17, 2026

Audit

Dependencies

No dependency data recorded yet.

Agent activity

32 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/NVIDIA/Megatron-Energon ↗

packagehttps://pypi.org/project/megatron-energon/ ↗