mongochangestream

library0.64.0jsnpmunverified

Sync MongoDB collections via change streams into any database. Version 0.64.0, requires Redis (>=5.4.1) for state management and MongoDB driver (>=6.8.0). Provides initial scan with deterministic resumption and change stream event processing. Detects schema changes and emits events (e.g., 'schemaChange', 'processError'). Supports batch processing and configurable queue/scan options. Compared to alternatives, it is a lower-level library meant to be built upon, with companion libraries for MongoDB-to-MongoDB, Elasticsearch, and CrateDB sync. Uses debug for logging. ESM-only, ships TypeScript types.

database

npm install mongochangestream

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · MONGOCHANGESTREAM

mongochangestream

databasejavascriptv0.64.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

initSync

✓ import { initSync } from 'mongochangestream'

✗ const initSync = require('mongochangestream');

ESM-only; CommonJS require will not work. The package does not export a default.

InitSyncResult

✓ import type { InitSyncResult } from 'mongochangestream'

Use type import for TypeScript when only type information is needed.

QueueOptions

✓ import type { QueueOptions } from 'mongochangestream'

TypeScript users should import types with 'import type'.

Shows how to initialize sync with Redis and MongoDB, run an initial scan, process change stream events, and detect schema changes with event handling.

import Redis from 'ioredis'; import { initSync } from 'mongochangestream'; import { MongoClient } from 'mongodb'; const redis = new Redis(); const mongoUrl = process.env.MONGO_URL ?? 'mongodb://localhost:27017'; const client = await MongoClient.connect(mongoUrl); const db = client.db('myDb'); const coll = db.collection('myColl'); const processCSRecords = async (docs) => { console.dir(docs, { depth: 10 }); }; const processRecords = async (docs) => { console.dir(docs, { depth: 10 }); }; const sync = initSync(redis, coll); const initialScan = await sync.runInitialScan(processRecords); initialScan.start(); const changeStream = await sync.processChangeStream(processCSRecords); changeStream.start(); setTimeout(changeStream.stop, 30000); const schemaChange = await sync.detectSchemaChange(db, { shouldRemoveMetadata: true, }); schemaChange.start(); sync.emitter.on('schemaChange', () => { initialScan.stop(); changeStream.stop(); });

Debug

Known issues

gotchaIf the Node process is stopped before the first change event is received, changes during restart may be missed.

fix

Implement a grace period or use the 'missingOplogEntry' utility to check if the change stream can resume from a stored token.

affects: >=0.0.0

gotchaThe initial scan sorts by _id by default; ensure your collection has an _id index for performance.

fix

Add an index on the _id field if not already present, or consider custom sorting if needed.

affects: >=0.0.0

deprecatedThe package does not export a default; importing default with 'import initSync from ...' will fail.

fix

Use named import: import { initSync } from 'mongochangestream'.

affects: >=0.0.0

gotchaRedis is required even for simple sync; if Redis is not available, the library will throw errors.

fix

Ensure Redis connection is properly configured. Run separate Redis instance or use ioredis with proper timeouts.

affects: >=0.0.0

Errors

Common errors & fixes

Error: The module 'mongochangestream' is not compatible with CommonJS. Use 'import' instead.

The package is ESM-only and cannot be required() in Node.js without special settings.

fix

Change require('mongochangestream') to a dynamic import or set type: 'module' in package.json.

TypeError: initSync is not a function

Likely imported default (import initSync from ...) instead of named import.

fix

Use import { initSync } from 'mongochangestream'.

MongoChangeStreamError: Redis required but not provided

Redis instance not passed to initSync or Redis connection issue.

fix

Ensure Redis client is created and passed: const redis = new Redis(); const sync = initSync(redis, coll);

MongoError: ChangeStream is not resumable: the resume token is no longer in the oplog

Oplog entry has aged out; change stream cannot resume from stored token.

fix

Use the missingOplogEntry() helper to detect this and resync from scratch.

Upgrade

Version history

0.64.0latest on npm

Audit

Dependencies

ioredisrequiredRequired for state management (full sync state, change stream resume tokens).

mongodbrequiredMongoDB driver for connecting to MongoDB and using ChangeStreamDocument types.

Agent activity

4 hits · last 30 days

node

Resources

githubgit://github.com/smartprocure/mongochangestream ↗

packagemongochangestream ↗

homepagehttps://github.com/smartprocure/mongochangestream#readme ↗