MPIRE is a Python package that simplifies multiprocessing, offering a faster and more user-friendly alternative to the standard `multiprocessing` module. It provides an intuitive API with map-like functions, support for worker state, progress bars (via tqdm), worker insights, and efficient handling of shared objects. The library is actively maintained with regular updates and aims to make parallelizing CPU-bound tasks straightforward and performant.
pip install mpireVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to use `mpire.WorkerPool` to parallelize a simple function across multiple processes. It initializes a pool with 4 workers and uses the `map` function, similar to Python's built-in `map` but executed in parallel.
Update your code to use `func` instead of `func_pointer` when calling `WorkerPool.map`, `map_unordered`, `imap`, or `imap_unordered`.
Remove the `restart_workers` parameter. Consider using `worker_lifespan` in the `WorkerPool` constructor for similar functionality of restarting workers after a specified number of tasks.
Pass `enable_insights=True` to the `WorkerPool` constructor instead of individual `map` calls.
On Windows, shared objects will be copied once per worker instead of using copy-on-write. This might still be more efficient than copying per task, but understand the performance implications compared to Linux/macOS.
Avoid mixing different types of `map` calls within the same `WorkerPool` instance. If you need to use different map styles, create separate `WorkerPool` instances for each.
These `OSError` messages can generally be safely ignored as they are related to internal cleanup processes.
If running on Windows and needing a progress bar, use a different start method (e.g., `spawn` or `forkserver` if available, though `fork` is not on Windows) or consider disabling the progress bar.
Define the function at the top-level of a module so it's globally accessible and picklable. Alternatively, set `use_dill=True` in the `WorkerPool` constructor (e.g., `with WorkerPool(use_dill=True) as pool:`) to use `dill` as the serialization backend, which can handle more complex Python objects, including local functions and lambdas.
Install the `mpire` package using pip: `pip install mpire`.
Install `ipywidgets` using pip (`pip install ipywidgets`) or conda (`conda install -c conda-forge ipywidgets`). If the issue persists, enable the Jupyter nbextension by running `jupyter nbextension enable --py --sys-prefix widgetsnbextension` and then restart your Jupyter notebook server.
When using lazy map functions (e.g., `imap`), ensure you iterate through all results to drain the queue. To prevent queues from becoming overly full and speed up shutdown, utilize the `max_tasks_active` parameter in the `WorkerPool` constructor (e.g., `with WorkerPool(max_tasks_active=n_jobs * 2) as pool:`).