Registry / testing / simplecrawler

simplecrawler

JSON →
library1.1.9jsnpmunverified

simplecrawler is an event-driven web crawler for Node.js (v1.1.9, stable, low release cadence). It provides flexible queue and cache mechanisms with extensible backends, automatic robots.txt respect, and link discovery. Differentiators vs alternatives like node-crawler or puppeteer: lightweight, uses EventEmitter, supports freezing/defrosting queues to disk, and preserves binary data via buffers. Suitable for archiving, analysis, and large-scale crawling.

testing
npm install simplecrawler
github ↗package ↗homepage ↗
INSTALL
IMPORT
SIG · SIMPLECRAWLER
S
simplecrawler
testingjavascriptv1.1.9
harness data pending
Install & Compatibility
Where this runs

No compatibility data collected yet for this library.

Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

Crawler
✓ const Crawler = require('simplecrawler'); const crawler = new Crawler('http://example.com');
Simplecrawler exports a constructor. You can omit 'new' and chain API calls.
Crawler (ESM)
✓ import Crawler from 'simplecrawler';
✗ import { Crawler } from 'simplecrawler';
Default import, not named. Works with ESM.
Queue
✓ const queue = crawler.queue;
✗ const { Queue } = require('simplecrawler');
Queue is not exported directly; access via crawler instance.

Basic crawler setup: configure interval and concurrency, register fetchcomplete and fetcherror events, then start crawling.

const Crawler = require('simplecrawler'); const crawler = new Crawler('https://example.com/'); crawler.interval = 1000; // milliseconds between requests crawler.maxConcurrency = 5; crawler.on('fetchcomplete', (queueItem, responseBuffer) => { console.log(`Fetched ${queueItem.url} (${responseBuffer.length} bytes)`); }); crawler.on('fetcherror', (queueItem, response) => { console.error(`Error fetching ${queueItem.url}: ${response.statusCode}`); }); crawler.start();
Debug
Known issues
gotchaQueue is not exported; must be accessed via crawler instance.
fix
Use crawler.queue instead of requiring a separate queue module.
affects: >=1.0.0
breakingMajor version 1 dropped support for Node < 8.
fix
Use Node >= 8 or stick to 0.x.
affects: >=1.0.0
deprecatedEvent names like 'fetchcomplete' are stable but not Promisified; use callbacks.
fix
No migration needed, but consider wrapping in promises if desired.
affects: >=1.0.0
gotchaDefault link discovery only detects same-domain links; cross-domain URLs are ignored.
fix
Set crawler.domainWhitelist to allow additional domains.
affects: >=1.0.0
gotchaAutomatic robots.txt parsing; if robots.txt is unreachable, crawling may be blocked.
fix
Set crawler.respectRobotsTxt = false to disable.
affects: >=1.0.0
Errors
Common errors & fixes
ReferenceError: Crawler is not defined
Missing require or incorrect import.
fix
const Crawler = require('simplecrawler');
Cannot find module 'simplecrawler'
Package not installed.
fix
npm install simplecrawler
TypeError: crawler.on is not a function
Crawler not instantiated; called on constructor directly.
fix
const crawler = new Crawler('http://example.com');
Upgrade
Version history
1.1.9latest on npm
Audit
Dependencies

No dependency data recorded yet.

Agent activity
4 hits · last 30 days
node
4
Resources
githubhttps://github.com/simplecrawler/simplecrawler ↗
packagesimplecrawler ↗
homepagehttps://github.com/simplecrawler/simplecrawler ↗
simplecrawler — npm install simplecrawler · libregistry