dbcp

library1.4.10jsnpmunverified

Database copy tool that streams data between MySQL, PostgreSQL, SQLServer, LevelDB, MongoDB, ElasticSearch, and cloud storage (S3, GCS, Azure) or files in JSON, ND-JSON, CSV, SQL, Parquet, and TFRecord formats. Current stable version is 1.4.10. Release cadence is irregular. Key differentiators: no intermediate storage required for streaming; supports sharding, compound inserts, and SQL dialect translation; built on Node.js streams with a CLI and TypeScript API.

database devops

npm install dbcp

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · DBCP

dbcp

databasejavascriptv1.4.10

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

dbcp

✓ import { dbcp } from 'dbcp'

✗ const dbcp = require('dbcp')

The package exports dbcp as a named export. Using default import or CommonJS require with named export will fail.

transformObject

✓ import { transformObject } from 'dbcp'

✗ import transformObject from 'dbcp'

transformObject is a named export within the dbcp function options, not a separate function.

AnyFileSystem

✓ import { AnyFileSystem } from '@wholebuzz/fs/lib/fs'

✗ import { AnyFileSystem } from 'dbcp'

AnyFileSystem is not exported by dbcp; it must be imported from @wholebuzz/fs/lib/fs.

Shows how to use the dbcp API to copy a local JSON file to a compressed NDJSON file on AWS S3, including file system setup.

import { AnyFileSystem } from '@wholebuzz/fs/lib/fs' import { LocalFileSystem } from '@wholebuzz/fs/lib/local' import { S3FileSystem } from '@wholebuzz/fs/lib/s3' import { dbcp } from 'dbcp' import StreamTree from 'tree-stream' const fileSystem = new AnyFileSystem([ { urlPrefix: 's3://', fs: new S3FileSystem() }, { urlPrefix: '', fs: new LocalFileSystem() } ]) // Copy a local JSON file to a compressed NDJSON file on S3 await dbcp({ fileSystem, inputFile: 'data.json', outputFile: 's3://bucket/data.jsonl.gz', })

Debug

Known issues

deprecatedThe CLI option '--inputStream' is deprecated in favor of '--inputFile'.

fix

Use --inputFile or --inputType instead.

affects: >=1.0.0

gotchaWhen using the API, fileSystem must be provided and must include all relevant file systems for the input/output URLs.

fix

Instantiate AnyFileSystem with the correct file system implementations for your cloud providers.

affects: >=1.0.0

breakingVersion 2.0.0 will require Node.js >=14 and may drop support for some database drivers.

fix

Upgrade Node.js and check database driver compatibility.

affects: >=2.0.0

gotchaThe transformObject option in the API is applied to each object in the stream; mutating the object can cause side effects.

fix

Create a new object if you need to modify fields.

affects: >=1.0.0

Errors

Common errors & fixes

Error: Cannot find module '@wholebuzz/fs'

@wholebuzz/fs is a peer dependency that is not installed automatically by npm.

fix

Run 'npm install @wholebuzz/fs'.

TypeError: dbcp is not a function

Using default import when dbcp is a named export.

fix

Use 'import { dbcp } from 'dbcp'' instead of 'import dbcp from 'dbcp''.

Error: inputFile or outputFile must be specified

Missing required input or output specification.

fix

Provide at least inputFile or outputFile, or both.

Upgrade

Version history

1.4.10latest on npm

Audit

Dependencies

knexrequiredUsed for database connections and queries

@wholebuzz/fsrequiredProvides file system abstraction for cloud and local storage

tree-streamrequiredProvides stream primitives (ReadableStreamTree, WritableStreamTree)

knex-schema-inspectoroptionalUsed for schema inspection

Agent activity

10 hits · last 30 days

node

Resources

githubhttps://github.com/wholebuzz/dbcp ↗

packagedbcp ↗

homepagehttps://github.com/wholebuzz/dbcp#readme ↗