khotan-data

library0.0.1jsnpmunverified

Data primitives for TypeScript — ETL pipelines, transforms, and Drizzle Postgres integration. Current stable version: 0.0.1. Pre-release, API likely unstable. Built for Next.js + Drizzle + Postgres stacks. Key differentiators: first-class Drizzle ORM support, typed pipelines, composable transforms, automatic batching for Postgres parameters, immutable Pipeline builder, and extractors for queries, paginated queries, and custom streams. Lightweight alternative to heavy ETL frameworks like Apache Beam or node-etl. Requires Node >= 18 and drizzle-orm >= 0.35.0 as a peer dependency.

database devops

npm install khotan-data

package ↗

INSTALL

IMPORT

SIG · KHOTAN-DATA

khotan-data

databasejavascriptv0.0.1

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Pipeline

✓ import { Pipeline } from 'khotan-data'

✗ const { Pipeline } = require('khotan-data')

ESM-only; no CommonJS support.

fromQuery

✓ import { fromQuery } from 'khotan-data'

✗ import { fromQuery } from 'khotan-data/drizzle'

fromQuery is re-exported from the main entry; drizzle-specific extractors are also available under 'khotan-data/drizzle' but the main barrel export works.

filter

✓ import { filter } from 'khotan-data'

✗ import { filter } from 'khotan-data/transform'

filter is re-exported from main; using the transform path is unnecessary but not incorrect.

map

✓ import { map } from 'khotan-data'

✗ import map from 'khotan-data'

Named export, not default export.

toDrizzleTx

✓ import { toDrizzleTx } from 'khotan-data'

✗ import { ToDrizzleTx } from 'khotan-data'

Function is camelCase, not PascalCase.

createExtractor

✓ import { createExtractor } from 'khotan-data'

Defines a typed ETL pipeline: extracts active users from Drizzle query, filters adults, transforms fields, then inserts into analytics table with batching.

import { Pipeline, fromQuery, map, filter, toDrizzle } from 'khotan-data'; import { db } from '@/db'; import { users, analytics } from '@/db/schema'; import { eq } from 'drizzle-orm'; const result = await Pipeline.create('user-analytics') .extract( fromQuery('active-users', () => db.select().from(users).where(eq(users.active, true)) ), ) .transform(filter('adults', (r) => r.age >= 18)) .transform( map('enrich', (r) => ({ userId: r.id, email: r.email.toLowerCase(), segment: r.age >= 65 ? 'senior' : 'standard', processedAt: new Date(), })), ) .load( toDrizzle('write-analytics', (rows) => db.insert(analytics).values(rows) ), ) .run(); console.log('Processed rows:', result.processed);

Debug

Known issues

breakingBreaking changes may occur before v1.0.0 as the API is pre-release (0.x).

fix

Lock version to a specific patch and test thoroughly on upgrades.

affects: >=0.0.0 <1.0.0

gotchaThe Pipeline builder is immutable: each method returns a new instance. Mutating or reusing a pipeline instance after calling .run() may lead to unexpected behavior.

fix

Always use the return value of .transform(), .load(), etc. Do not reuse a Pipeline instance after calling .run().

affects: >=0.0.0

gotchafromQueryPaginated and fromQueryCursor require an async generator factory, not a plain promise. Passing a synchronous function will cause runtime errors.

fix

Ensure query factory returns a Promise or is async: (limit, offset) => db.select()...

affects: >=0.0.0

deprecatedThe continueOnError option is experimental and may change in future versions.

fix

Avoid relying on it for production; test error handling explicitly.

affects: >=0.0.0

gotchatoDrizzle automatically batches inserts to stay under Postgres parameter limits, but the batch size calculation may be off for tables with many columns (>1600) leading to errors.

fix

Manually set columnsPerRow option in toDrizzle for very wide tables.

affects: >=0.0.0

Errors

Common errors & fixes

Error: Could not resolve 'khotan-data/drizzle'

Import from subpath without proper exports field or using bundler without support.

fix

Use 'khotan-data' main export instead; drizzle symbols are re-exported there.

TypeError: db is not a function

Passing a Drizzle instance directly to fromQuery instead of a callback.

fix

Wrap the query in a function: fromQuery('name', () => db.select()...)

Cannot find module 'khotan-data' or its corresponding type declarations.

Missing peer dependency drizzle-orm or Node version <18.

fix

Ensure drizzle-orm >=0.35.0 is installed and Node >=18.

The 'run()' method cannot be called more than once on the same Pipeline instance.

Pipeline is not designed for reuse; once run, internal state may be consumed.

fix

Create a new Pipeline for each execution: const p = Pipeline.create(...); await p.run();

Upgrade

Version history

0.0.1latest on npm

Audit

Dependencies

drizzle-ormrequiredCore integration for Drizzle extractors and loaders

Agent activity

6 hits · last 30 days

node

Resources

packagekhotan-data ↗