mongo2crate

library0.60.0jsnpmunverified

Sync MongoDB collections to CrateDB and convert JSON schemas to SQL DDL. Version 0.60.0, actively maintained. Provides two main workflows: incremental sync using MongoDB change streams with initial scan, and schema conversion to CREATE TABLE statements. Key differentiators include support for schema change detection, custom value mapping (e.g., string truncation), and field omission. Ships TypeScript types. Requires peer dependencies ioredis (for change stream cursor persistence) and mongodb driver.

database

npm install mongo2crate

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · MONGO2CRATE

mongo2crate

databasejavascriptv0.60.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

initSync

✓ import { initSync } from 'mongo2crate'

✗ const { initSync } = require('mongo2crate')

Package ships ESM. CommonJS require will fail unless using Node.js with ESM interop.

convertSchema

✓ import { convertSchema } from 'mongo2crate'

✗ import { convertSchema } from 'mongo2crate/schema'

convertSchema is exported at the package root, not from a subpath.

crate

✓ import { crate } from 'mongo2crate'

✗ import crate from 'mongo2crate'

crate is a named export, not a default export.

Shows how to connect to MongoDB and CrateDB, initialize sync with Redis, create table from schema, and process change stream with initial scan.

import { MongoClient } from 'mongodb' import { default as Redis } from 'ioredis' import { crate, initSync, convertSchema } from 'mongo2crate' const client = await MongoClient.connect(process.env.MONGO_URL ?? '') const db = client.db() const sync = initSync( new Redis({ keyPrefix: 'cratedb:' }), db.collection('myCollection'), crate({ url: process.env.CRATE_URL ?? '' }), { omit: ['secretField'], mapper: (node) => node.val } ) // Create table from schema const schema = await sync.getCollectionSchema(db) if (schema) { await sync.createTableFromSchema(schema) } // Start change stream const changeStream = await sync.processChangeStream() changeStream.start() // Initial scan const options = { batchSize: 1000 } const initialScan = await sync.runInitialScan(options) initialScan.start()

Debug

Known issues

breakinginitSync() now requires an explicit Redis instance for cursor persistence. Previously it defaulted to an in-memory store.

fix

Pass a Redis instance as first argument to initSync(). Example: initSync(new Redis({ keyPrefix: 'cratedb:' }), collection, crate(), options).

affects: >=0.50.0

deprecatedsync.createTableFromSchema() without prior getCollectionSchema() call will throw. Schema argument is no longer optional.

fix

Always call sync.getCollectionSchema(db) first and pass the returned schema to createTableFromSchema().

affects: >=0.55.0

gotchaString fields are truncated to 250 characters by default unless a mapper function is provided. CrateDB has a 32k limit but default columnar index imposes a lower limit.

fix

Provide a custom mapper function that slices strings as needed: mapper: (node) => typeof node.val === 'string' ? node.val.slice(0, 250) : node.val

affects: >=0.10.0

gotchachangeStream.stop() is emitted as an event; calling it directly will cause undefined behavior. Always listen on the emitter and call stop via event handler.

fix

Use sync.emitter.on('schemaChange', changeStream.stop) instead of calling changeStream.stop() directly.

affects: >0.0.0

gotchaMongoDB connection must be established before calling sync.getCollectionSchema(db). Passing undefined or null for db results in runtime error.

fix

Always await MongoClient.connect() and pass the resolved db object to getCollectionSchema().

affects: >0.0.0

Errors

Common errors & fixes

TypeError: Cannot read properties of undefined (reading 'collection')

The first argument to initSync is not a valid Redis client or is undefined.

fix

Ensure Redis client is properly instantiated: new Redis({ ... }) and passed as first argument.

Error: Schema is required

Calling createTableFromSchema() without a valid schema object.

fix

Call getCollectionSchema(db) first and pass its return value to createTableFromSchema().

SyntaxError: Cannot use import statement outside a module

Package is ESM-only; using require() or running in a CommonJS environment.

fix

Use import syntax and ensure project is configured as ESM (type: module in package.json) or use dynamic import.

MongoChangeStreamError: ns not found in oplog for collection myCollection

Change streams require a replica set or sharded cluster. Running on a standalone MongoDB instance.

fix

Use a MongoDB replica set or use the 'watch' option with { fullDocument: 'updateLookup' } after ensuring replica set is enabled.

Upgrade

Version history

0.60.0latest on npm

Audit

Dependencies

ioredisrequiredRequired for persisting change stream cursor state across restarts.

mongodbrequiredRequired for connecting to MongoDB and accessing change streams.

Agent activity

2 hits · last 30 days

node

Resources

githubgit://github.com/smartprocure/mongo2crate ↗

packagemongo2crate ↗

homepagehttps://github.com/smartprocure/mongo2crate#readme ↗