mongo2elastic

library0.58.0jsnpmunverified

A TypeScript library (v0.58.0) that syncs MongoDB collections to Elasticsearch using change streams and optional schema conversion. It supports initial bulk scans, real-time change stream processing, schema change detection, and automatic mapping generation from MongoDB JSON schemas with Elasticsearch-specific overrides (e.g., geo_point, nested types, copy_to). Active development with monthly releases. Key differentiators: provides both a sync engine and standalone schema conversion utilities, handles change stream lifecycle (start/stop on schema change), and supports Redis-backed resume tokens. Requires Node >=16, MongoDB >=6.8, Elasticsearch >=8.14, and ioredis >=5.4.

storage devops

npm install mongo2elastic

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · MONGO2ELASTIC

mongo2elastic

storagejavascriptv0.58.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

initSync

✓ import { initSync } from 'mongo2elastic'

✗ const { initSync } = require('mongo2elastic')

ESM-only; CJS require() may fail due to lack of default export.

convertSchema

✓ import { convertSchema } from 'mongo2elastic'

✗ import convertSchema from 'mongo2elastic'

convertSchema is a named export, not default. Also available as part of the library.

copyTo

✓ import { copyTo, isStringLike, setESType } from 'mongo2elastic'

✗ import { CopyTo } from 'mongo2elastic' (case-sensitive)

Helper functions are lowercase named exports. TypeScript types are included.

crate

✓ import { crate, initSync } from 'mongo2elastic'

✗ import { Crate } from 'mongo2elastic'

crate is a named export; likely the engine class. Check docs for usage.

Shows a full sync setup: MongoDB connection, Elasticsearch client, Redis for resume tokens, optional field omission, index creation with ignore_malformed, mapping from schema, change stream processing, schema change detection, and initial scan.

import { MongoClient } from 'mongodb'; import elasticsearch from '@elastic/elasticsearch'; import { default as Redis } from 'ioredis'; import { crate, initSync } from 'mongo2elastic'; import retry from 'p-retry'; const client = await MongoClient.connect(process.env.MONGO_CONN ?? ''); const db = client.db(); const elastic = new elasticsearch.Client({ node: process.env.ELASTIC_NODE ?? '', auth: { username: process.env.ELASTIC_USERNAME ?? '', password: process.env.ELASTIC_PASSWORD ?? '' } }); const sync = initSync(new Redis(), db.collection('myCollection'), elastic, { omit: ['password', 'unneededStuff'], }); sync.emitter.on('process', console.info); sync.emitter.on('error', console.error); sync.emitter.on('cursorError', () => process.exit(1)); await sync.createIndexIgnoreMalformed().catch(console.warn); const schema = await sync.getCollectionSchema(db); if (schema) { await sync.createMappingFromSchema(schema); } const changeStream = await sync.processChangeStream(); changeStream.start(); const schemaChange = await sync.detectSchemaChange(db); schemaChange.start(); sync.emitter.on('schemaChange', changeStream.stop); const options = { batchSize: 1000 }; const initialScan = await sync.runInitialScan(options); initialScan.start();

Debug

Known issues

breakingRequires Node >=16 and peer dependencies with strict semver ranges. Using older versions will fail to install or run.

fix

Ensure Node.js >=16, MongoDB driver >=6.8, Elasticsearch client >=8.14, ioredis >=5.4.

affects: >=0.58.0

breakingESM-only package. Attempting to require() via CommonJS will throw at runtime (ERR_REQUIRE_ESM).

fix

Use ES module syntax (import/export) and set 'type: module' in package.json, or use dynamic import() in a CJS context.

affects: >=0.58.0

gotchainitSync requires a Redis instance for resume tokens. Omitting or using a fake Redis will cause change stream failures on restart.

fix

Provide a valid ioredis instance. If not needed, check docs for alternative initialization.

affects: >=0.0.0

gotchaThe change stream will stop if a schema change is detected (via schemaChange event). Ensure you handle restarts or reinitialization.

fix

Listen for 'schemaChange' and re-run schema detection and mapping creation as needed.

affects: >=0.0.0

gotchacreateIndexIgnoreMalformed() may fail if the index already exists. It's wrapped in a .catch() in the example, but you may need to delete or update the index manually.

fix

Either delete the existing index first or handle the error gracefully.

affects: >=0.0.0

Errors

Common errors & fixes

ERR_REQUIRE_ESM: Must use import to load ES Module: /path/to/node_modules/mongo2elastic/index.js

Package is ESM-only but imported via require() in CommonJS.

fix

Use import { initSync } from 'mongo2elastic' or set type: module in package.json.

TypeError: redisInstance is not a constructor

initSync first argument expects a Redis instance (not a factory or client).

fix

Instantiate Redis via new Redis() before passing to initSync.

Error: MongoError: Cannot use change streams on a collection without a replica set

MongoDB change streams require a replica set. Running on standalone instance fails.

fix

Connect to a MongoDB replica set or use a service like MongoDB Atlas.

Elasticsearch client 8.x auth failure: ConnectionError: [connection_error] connect ECONNREFUSED

Invalid Elasticsearch endpoint or authentication credentials.

fix

Check ELASTIC_NODE, ELASTIC_USERNAME, and ELASTIC_PASSWORD environment variables. Ensure Elasticsearch is running.

Upgrade

Version history

0.58.0latest on npm

Audit

Dependencies

@elastic/elasticsearchrequiredpeer dependency for Elasticsearch client v8+

ioredisrequiredpeer dependency for Redis-backed resume tokens in change stream processing

mongodbrequiredpeer dependency for MongoDB driver v6+

Agent activity

21 hits · last 30 days

node

Resources

githubgit://github.com/smartprocure/mongo2elastic ↗

packagemongo2elastic ↗

homepagehttps://github.com/smartprocure/mongo2elastic#readme ↗