Registry / data / modin

modin

JSON →
library0.37.1pypypi✓ verified 35d ago

Modin is an open-source Python library designed to accelerate pandas workflows by leveraging distributed computing frameworks like Ray, Dask, or Unidist. It aims to be a drop-in replacement for pandas, requiring only a single line change to the import statement. Modin supports datasets ranging from MBs to TBs, offering significant speedups, especially for larger data, and is actively maintained with frequent releases.

datadevopsworkflow
Install & Compatibility
Where this runs
tested against v0.37.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
glibc
py 3.10
5/20 runs
18/20 runs
py 3.11
5/20 runs
19/20 runs
py 3.12
5/20 runs
19/20 runs
py 3.13
5/20 runs
15/20 runs
py 3.9
5/20 runs
✓ 20.56s
762MB installed
● package 762MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

To leverage Modin's accelerated functionality, replace the standard pandas import with `modin.pandas`.

import modin.pandas as pd

This quickstart demonstrates how to use Modin by simply changing the pandas import statement. Modin automatically parallelizes DataFrame operations across available cores/nodes using the installed backend (Ray, Dask, or Unidist). For small datasets, the performance difference might be negligible or even slightly worse due to overhead, but it offers significant speedups for large data.

import modin.pandas as pd # Create a Modin DataFrame (this uses your configured backend, e.g., Ray or Dask) data = {'col1': [1, 2, 3, 4], 'col2': [5, 6, 7, 8]} df = pd.DataFrame(data) print("Original Modin DataFrame:") print(df) # Perform a common operation, like adding a new column df['col3'] = df['col1'] + df['col2'] print("DataFrame after operation:") print(df) # Note: For larger datasets, Modin's performance benefits become more apparent. # Example with a larger dataset (requires a CSV file, e.g., 'nyc_taxi_data.csv') # Uncomment and replace 'path/to/your/data.csv' with an actual path to test with large data. # try: # large_df = pd.read_csv('path/to/your/data.csv') # print(f"Loaded large DataFrame with {len(large_df)} rows.") # print(large_df.head()) # except FileNotFoundError: # print("Skipping large dataset example: 'path/to/your/data.csv' not found.")
Debug
Known footguns
gotchaNot all pandas operations are fully implemented in Modin. When an unimplemented method is called, Modin may silently fall back to the single-threaded pandas implementation, incurring communication overhead and potentially being slower than native pandas. A `UserWarning` might be issued.
gotchaMixing pandas and Modin DataFrames directly in the same workflow is not recommended. Passing a pandas DataFrame to a Modin method or vice versa can lead to performance degradation (due to conversion overhead) or undefined behavior, as pandas identifies Modin objects as simple iterables.
gotchaFor very small datasets (MBs), the overhead introduced by Modin's distributed computing engine (Ray, Dask, etc.) can sometimes make operations slower than plain pandas. Modin is optimized for medium to large datasets (GBs to TBs).
breakingSince Modin 0.30.0, the default Ray installation uses `ray-core` instead of `ray-default`. This means the Ray dashboard and cluster launcher are no longer installed by default.
breakingAs of Modin 0.31.0, the HDK engine and Cudf storage format have been removed, as they were unmaintained.
gotchaModin's `read_csv` function might not handle certain edge cases or exceptions as gracefully as native pandas. This can lead to errors (e.g., `TypeError` for missing columns) in situations where pandas would succeed.
gotchaErrors like `ArrowIOError: Broken Pipe` or Modin 'hanging on import' can occur if the underlying Ray (or Dask) backend fails to start correctly or unexpectedly shuts down (e.g., due to `KeyboardInterrupt`, system sleep, or rapid restarts of notebooks).
Upgrade
Version history

Breaking-change detection hasn't run for this library yet.

Audit
Security & dependencies

CVE tracking and dependency tree are planned for a later release.

Agent activity
18 hits · last 30 days
gptbot
4
ahrefsbot
4
bytedance
4
dotbot
1
script
1
Resources