Registry / database / wikipedia-to-mongodb

wikipedia-to-mongodb

library2.4.0jsnpmunverified

Imports a full Wikipedia XML dump into MongoDB, parsing wikiscript into queryable JSON. Version 2.4.0. Release cadence: sporadic, last update 2017. One-liner from CLI or Node.js script. Uses wtf_wikipedia for parsing. Works with any language Wikipedia dump. Supports plaintext extraction and optional Redis-backed workers for faster loading. Requires Node.js ≥0.10.33 and MongoDB.

database

npm install wikipedia-to-mongodb

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · WIKIPEDIA-TO-MONGO

wikipedia-to-mongodb

databasejavascriptv2.4.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

default

✓ const wp2mongo = require('wikipedia-to-mongodb')

✗ import wp2mongo from 'wikipedia-to-mongodb'

Only CommonJS supported. No named exports; use default require.

wp2mongo (with callback)

✓ wp2mongo({file: 'dump.xml.bz2', db: 'mydb'}, callback)

✗ wp2mongo({file: 'dump.xml.bz2'})

db option defaults to 'enwiki' if omitted. Callback is optional but recommended.

wp2mongo (with promise)

✓ wp2mongo({file: 'dump.xml.bz2', db: 'mydb'}).then(/*...*/)

✗ await wp2mongo({file: 'dump.xml.bz2'})

Returns a promise only if callback not provided. Mixing callback and promise is undefined.

Imports a Wikipedia XML dump into MongoDB with options for plaintext extraction and redirect handling.

const wp2mongo = require('wikipedia-to-mongodb'); wp2mongo({ file: './enwiki-latest-pages-articles.xml.bz2', db: 'enwiki', plaintext: true, skip_redirects: false }, (err, stats) => { if (err) return console.error('Error:', err.message); console.log('Imported', stats.pages, 'pages in', stats.time, 'seconds'); console.log('Collection name:', stats.collection); });

Debug

Known issues

deprecatedNode.js 0.10.x is deprecated. Use newer Node.js at your own risk.

fix

Upgrade to Node.js 4+ or use older version of package.

affects: >=2.0.0

gotchaLarge dumps (like enwiki) can take hours and produce multi-GB collections. Ensure sufficient disk space and RAM.

fix

Use the --worker option with Redis to parallelize and speed up import.

affects: *

gotchaThe `skip_redirects` option defaults to `false`, meaning redirect pages are included. This may double the document count.

fix

Set `skip_redirects: true` to exclude redirects.

affects: *

gotchaNo schema validation or index creation is performed by default. Queries may be slow on large collections.

fix

Create indexes manually after import: `db.wikipedia.createIndex({title:1})`

affects: *

gotchaThe package uses streams but does not handle backpressure perfectly. Memory usage may spike with large dumps.

fix

Use the --worker option to offload parsing to child processes.

affects: *

Errors

Common errors & fixes

Error: Cannot find module 'wtf_wikipedia'

Missing dependency when installing via npm -g or without --save.

fix

Run `npm install -g wikipedia-to-mongodb wtf_wikipedia` or `npm install --save wikipedia-to-mongodb`

MongoError: connect ECONNREFUSED 127.0.0.1:27017

MongoDB not running or not reachable.

fix

Start MongoDB with `mongod` or specify a different host/port in connection string.

Error: ENOENT: no such file or directory, open '/path/to/dump.xml.bz2'

Dump file path is incorrect or missing.

fix

Provide the correct absolute or relative path to the .xml.bz2 file.

Upgrade

Version history

2.4.0latest on npm

Audit

Dependencies

wtf_wikipediarequiredparses wikiscript to JSON

Agent activity

29 hits · last 30 days

node