Registry / devops / node-worker-pool

node-worker-pool

JSON →
library3.0.2jsnpmunverified

node-worker-pool is a JavaScript library for Node.js that provides a robust mechanism for managing a pool of child worker processes. It's designed for scenarios involving numerous highly parallelizable tasks, utilizing an exclusive message-passing paradigm rather than shared memory for inter-process communication. The library, currently at version 3.0.2, allows developers to define custom worker executables (though helper libraries are currently Node.js-specific) that communicate over `stdin`/`stdout` using a defined protocol. While its release cadence isn't explicitly documented, it focuses on stability and efficient task distribution. Key differentiators include its explicit message-passing design and the ability to use external executables as workers, offering flexibility beyond typical Node.js `worker_threads` when process isolation and custom language workers are desired.

devopsworkflow
npm install node-worker-pool
github ↗package ↗
INSTALL
IMPORT
SIG · NODE-WORKER-POOL
N
node-worker-pool
devopsjavascriptv3.0.2
Install
—
Import
—
Disk
—
Pass rate
0/ 6
Env Coverage0 / 6
glibc
18–22
musl
18–22
Install & Compatibility
Where this runs
tested against v? · npm install
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
node 18–226 runs
build_error
glibc
node 18–226 runs
build_error
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

WorkerPool
✓ const WorkerPool = require('node-worker-pool');
✗ import WorkerPool from 'node-worker-pool';
The library primarily uses CommonJS `require` syntax. As of v3, ESM `import` is not officially supported and will likely result in a runtime error.
workerUtils
✓ const workerUtils = require('node-worker-pool/nodeWorkerUtils');
✗ import * as workerUtils from 'node-worker-pool/nodeWorkerUtils';
Specific utilities for implementing a Node.js worker are exposed via a sub-path import, strictly using CommonJS `require`.
{ startWorker, respondWithError }
✓ const { startWorker, respondWithError } = require('node-worker-pool/nodeWorkerUtils');
✗ import { startWorker, respondWithError } from 'node-worker-pool/nodeWorkerUtils';
Functions like `startWorker` and `respondWithError` are named exports from `nodeWorkerUtils` and must be destructured using CommonJS `require`.

This quickstart demonstrates how to set up a `WorkerPool` with a dynamically created worker script, send messages to the pool, and process responses, showing inter-process communication.

const fs = require('fs'); const path = require('path'); const WorkerPool = require('node-worker-pool'); const workerScriptContent = ` var workerUtils = require('node-worker-pool/nodeWorkerUtils'); var initData; function onInitialize(data) { initData = data; } function onMessage(data) { return { initData: initData, receivedData: data, pid: process.pid }; } if (require.main === module) { try { workerUtils.startWorker(onInitialize, onMessage); } catch (e) { workerUtils.respondWithError(e); } } `; // Create a temporary worker file const workerFilePath = path.join(__dirname, 'temp-worker.js'); fs.writeFileSync(workerFilePath, workerScriptContent); async function runPool() { console.log('Starting worker pool...'); const workerPool = new WorkerPool( 2, // Use 2 workers for demonstration process.execPath, // Path to the node binary workerFilePath, // Path to the temporary worker script { initData: { someUsefulConstant: 42 } } ); try { console.log('Sending messages to workers...'); const responses = await Promise.all([ workerPool.sendMessage({ message: 'Hello from main 1!' }), workerPool.sendMessage({ message: 'Hello from main 2!' }), workerPool.sendMessage({ message: 'Hello from main 3!' }) ]); responses.forEach((response, index) => { console.log(`Response ${index + 1}:`, response); }); console.log('Shutting down worker pool...'); await workerPool.shutDown(); console.log('All worker processes have now been killed.'); } catch (error) { console.error('An error occurred:', error); } finally { // Clean up the temporary worker file fs.unlinkSync(workerFilePath); process.exit(0); } } runPool();
Debug
Known issues
gotchaThe communication protocol between the main process and workers is not officially documented, making it challenging to implement workers in languages other than Node.js or to debug complex communication issues.
fix
Adhere strictly to the `nodeWorkerUtils` helper library for Node.js workers. For non-Node.js workers, reverse-engineer the `stdin`/`stdout` messaging protocol by inspecting `node-worker-pool`'s source code.
affects: >=1.0.0
gotchaWorker functions (`onInitialize`, `onMessage`) must only return plain JavaScript objects. Returning primitive values (strings, numbers, booleans) or `null`/`undefined` from `onMessage` will lead to a 'Worker protocol error: Response must be an object'.
fix
Always ensure the return value of `onMessage` is a serializable object. Wrap any primitive return values in an object, e.g., `return { result: 'your_value' };`.
affects: >=1.0.0
gotchaThe `workerExecutablePath` and `workerScriptPath` arguments to `WorkerPool` must be correct and accessible from the main process. Incorrect paths will result in worker startup failures that might be difficult to diagnose.
fix
Use absolute paths or paths relative to `process.cwd()` for both `process.execPath` (for the Node.js binary) and your worker script path (e.g., `path.join(__dirname, 'worker.js')`). Implement robust error handling around worker startup.
affects: >=1.0.0
Errors
Common errors & fixes
Error: Cannot find module 'node-worker-pool'
The `node-worker-pool` package is not installed or not accessible from the current working directory.
fix
Run `npm install node-worker-pool` or `yarn add node-worker-pool` in your project directory. Ensure your Node.js module resolution paths are correctly configured.
ReferenceError: WorkerPool is not defined
The `WorkerPool` class was used without being correctly `require`d into the scope.
fix
Add `const WorkerPool = require('node-worker-pool');` at the top of your file where `WorkerPool` is used.
Worker protocol error: Response must be an object
The `onMessage` function in the worker script returned a value that was not a plain JavaScript object.
fix
Modify your `onMessage` function in the worker script to always return an object, e.g., `return { data: yourResult };`.
Error: spawn [worker executable path] ENOENT
The path provided for the worker executable (e.g., `process.execPath` or a custom binary) or the worker script itself is incorrect or the file does not exist.
fix
Verify that `process.execPath` correctly points to your Node.js binary and that the `workerScriptPath` argument correctly points to your worker script file. Use absolute paths for robustness.
Upgrade
Version history
3.0.2latest on npm
Audit
Dependencies

No dependency data recorded yet.

Agent activity
18 hits · last 30 days
node
16
OpenAI (training)
1
Resources
githubhttps://github.com/jeffmo/node-worker-pool ↗
packagehttps://www.npmjs.com/package/node-worker-pool ↗
node-worker-pool — npm install node-worker-pool · libregistry