billiard is a robust fork of the Python 2.7 `multiprocessing` package, actively maintained by the Celery project. It provides numerous improvements and bugfixes over the standard library's `multiprocessing` module, offering enhanced process-based parallelism for Python applications. It aims to address specific challenges and performance bottlenecks, particularly in distributed task queue systems like Celery, where it serves as a core dependency. The library is under active development with a consistent release cadence.
pip install billiardVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to create and use a process pool with `billiard.Pool` to execute a function across multiple processes. The `if __name__ == '__main__':` guard is essential for proper functioning, especially on Windows and with certain start methods, to prevent infinite process spawning.
Upgrade your Python environment to 3.7 or higher.
Ensure your code is 'spawn'-safe (i.e., doesn't rely on inherited state, and uses the `if __name__ == '__main__':` guard). If 'fork' is strictly necessary and you understand the risks, configure the start method explicitly (e.g., `billiard.set_start_method('fork')`).Always wrap the code that creates `Process` or `Pool` objects and their associated logic within an `if __name__ == '__main__':` block.
If parallel execution is needed within a Celery task, consider restructuring the workflow to use Celery's native group/chain/chord primitives or ensure that any internal parallelization is carefully managed to avoid conflicts with `billiard`'s process management.
Review any custom signal handling logic and remove reliance on `SIGUSR2` for process termination if you are using `billiard`'s default signal sets.
Install the 'billiard' library using pip: `pip install billiard` or `pip3 install billiard` if you have multiple Python versions.
Ensure that all objects passed to 'billiard' processes (e.g., as arguments to a `Process` target function or elements in a `Queue`) are picklable. This often involves defining functions at the top level of a module, making classes and their methods independently picklable, or restructuring the data to remove unpicklable components.
Redesign the application to avoid nesting process creation. If a background task needs to perform concurrent operations, it should use threads instead of creating new processes. In a Celery context, this means ensuring your tasks do not themselves invoke `billiard.Process` or `multiprocessing.Process`.
Inspect detailed worker logs for the specific exception or signal that led to the premature exit. Increase system memory or adjust resource limits if out-of-memory errors are indicated. Implement robust error handling within your tasks and configure Celery's `task_soft_time_limit` and `task_time_limit` settings appropriately.