Bounded Pool Executor is a Python library that provides `BoundedThreadPoolExecutor` and `BoundedProcessPoolExecutor` classes, extending `concurrent.futures` to manage a fixed-size queue for tasks. This prevents memory exhaustion that can occur with the standard unbounded queues in `concurrent.futures` when submitting a large number of tasks. The current version is 0.0.3, offering a solution to prevent memory leaks in high-concurrency scenarios by blocking `submit` calls when the queue is full.
pip install bounded-pool-executorVerified import paths — ran on the pinned version, not inferred.
This example demonstrates how to use `BoundedThreadPoolExecutor` to execute tasks while limiting the number of concurrent tasks and the size of the waiting queue. The `submit` method will block if the queue, including currently executing tasks, reaches its `max_queue_size`.
Manually add `add_done_callback` to futures to check for exceptions: `future.add_done_callback(lambda f: print(f'Task error: {f.exception()}') if f.exception() else None)`.Set `max_queue_size` to `max_workers + M`, where `M` is the desired number of tasks that can be waiting in the queue. A value of at least `2 * max_workers` is often recommended to keep workers busy.
Ensure that any function or object passed to `BoundedProcessPoolExecutor.submit` is top-level (not nested) and picklable. Avoid passing `self` from a class instance that also holds the `ProcessPoolExecutor`.
Design tasks to be independent. If tasks need to submit sub-tasks, consider a separate, dedicated pool for sub-tasks, or ensure sufficient pool and queue capacity to prevent circular dependencies that lead to deadlock.
Ensure the import statement uses the correct module name `bounded_pool`. If not already installed, first run `pip install bounded-pool`. `from bounded_pool import BoundedThreadPoolExecutor`
Instead of `executor.map(func, iterable)`, use a loop with `executor.submit` and collect the `Future` objects, then retrieve results using `future.result()` or `concurrent.futures.as_completed`.
```python
from bounded_pool import BoundedThreadPoolExecutor
def my_task(item):
return item * 2
items =
results = []
with BoundedThreadPoolExecutor(max_workers=2, max_queue_size=2) as executor:
futures = [executor.submit(my_task, item) for item in items]
for future in futures:
results.append(future.result())
print(results)
```First, install the package using pip: `pip install bounded-pool`. Then, ensure the import statement is `from bounded_pool import BoundedThreadPoolExecutor` (or `BoundedProcessPoolExecutor`).
No dependency data recorded yet.