aiomultiprocess is a Python library that provides an asynchronous version of the standard `multiprocessing` module, combining the benefits of `asyncio` for I/O-bound tasks and `multiprocessing` for CPU-bound tasks. It runs a full `asyncio` event loop on each child process, enabling high levels of concurrency and parallelism beyond the Global Interpreter Lock (GIL). The library is actively maintained, with its current version being 0.9.1.
pip install aiomultiprocessVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates using `aiomultiprocess.Pool` to fetch multiple URLs concurrently across several processes. It defines an asynchronous function `fetch_url_content` that uses `aiohttp` to make an HTTP GET request. The `main` function then creates a `Pool` and uses its `map` method to distribute the `fetch_url_content` coroutine calls across the worker processes, iterating over results as they complete.
Ensure all functions, classes, and global objects passed to worker processes are defined at the top-level of a module and are importable. If 'forked' behavior is strictly required (e.g., for sharing non-pickleable resources that are copied by fork), call `aiomultiprocess.set_start_method('fork')` before creating any workers or pools.Avoid relying on shared mutable global state. Use queues (`multiprocessing.Manager().Queue()`) for explicit inter-process communication or pass data explicitly as arguments and return values. For resource initialization per process, use the `initializer` and `initargs` parameters of the `Pool` constructor.
To handle exceptions within the worker process itself, provide an `exception_handler` callable to the `Pool` (or `Process`/`Worker`) constructor. This handler will be called with the exception object before it's propagated back to the main process.
Utilize the `maxtasksperchild` parameter when creating a `Pool`. Setting it to a positive integer will cause worker processes to exit and be respawned after completing the specified number of tasks, helping to release resources and prevent file handle exhaustion.
Explicitly specify the event loop initializer using the `loop_initializer` parameter in the `Pool` (or `Process`/`Worker`) constructor. For `uvloop`, this would be `loop_initializer=uvloop.new_event_loop`.
Install the package using pip: 'pip install aiomultiprocess'.
Ensure that the 'main()' coroutine is properly awaited by using 'asyncio.run(main())' in Python 3.7+, or 'loop = asyncio.get_event_loop(); loop.run_until_complete(main())' in Python 3.6.
Install the 'aiohttp' package using pip: 'pip install aiohttp'.
Ensure that functions and objects intended for use in child processes are defined at the top-level of a module, making them importable by the new process. Avoid using `lambda` functions or nested function definitions as targets for multiprocessing tasks. If passing complex objects, ensure they are serializable or use an initializer function for setup within the child processes.
Ensure that `asyncio.run()` or `loop.run_until_complete()` is called only once per thread/process to manage the top-level event loop. In child processes managed by `aiomultiprocess`, the event loop is usually set up automatically; avoid explicitly calling `asyncio.run()` within the target functions executed by the pool or worker, unless you are deliberately managing separate loops in a highly specific way.