crawlex

library1.0.4jsnpmunverified

A stealth web crawler for Node.js (≥20) that achieves Chrome-perfect TLS and HTTP/2 fingerprinting, making bot detection nearly impossible. v1.0.4 (April 2025) is actively maintained with weekly releases focused on stability and feature parity. It includes a browser render pool for JavaScript-heavy pages, persistent queue management, lifecycle hooks, and built-in sitemap support. Unlike Playwright or Puppeteer, CrawleX provides a higher-level orchestration layer optimized for large-scale scraping with anti-bot evasion as a core feature. Ships TypeScript types and is ESM-only. Key differentiators: automatic fingerprint rotation, adaptive throttling, and a plugin system for custom middleware.

devops

npm install crawlex

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · CRAWLEX

crawlex

devopsjavascriptv1.0.4

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

crawlex

✓ import crawlex from 'crawlex'

✗ const crawlex = require('crawlex')

CrawleX is ESM-only and does not support CommonJS require. Default export is the main crawler function.

CrawleXConfig

✓ import type { CrawleXConfig } from 'crawlex'

✗ import { CrawleXConfig } from 'crawlex'

Config type is only available as a type import. Using a value import will cause a runtime error in TypeScript.

HookType

✓ import { HookType } from 'crawlex'

Named export for hook enum used in lifecycle hooks. Works with ESM import.

CrawlQueue

✓ import { CrawlQueue } from 'crawlex'

✗ import { Queue } from 'crawlex'

The persistent queue class is exported as CrawlQueue, not Queue.

Demonstrates basic usage of CrawleX with a rendering pool, stealth mode, hooks, and progress reporting.

import crawlex from 'crawlex'; const results = await crawlex({ urls: ['https://example.com', 'https://httpbin.org/html'], followLinks: true, maxDepth: 2, concurrency: 5, renderJavaScript: true, // use headless browser for JS headless: true, // run in headless mode (default true) stealth: true, // enable TLS/H2 fingerprint spoofing hooks: { onRequest: (url) => console.log(`Crawling: ${url}`), onResponse: (response) => console.log(`Status: ${response.status}`), onError: (url, error) => console.error(`Error: ${url} - ${error.message}`) }, onProgress: (progress) => console.log(`Progress: ${progress.completed}/${progress.total}`) }); console.log(`Collected ${results.length} pages`);

Debug

Known issues

breakingNode.js >=20 is required; lower versions will throw engine errors.

fix

Upgrade Node.js to version 20 or later.

affects: <1.0.0

deprecatedThe option 'headless' is deprecated in favor of 'headless: true' (which is the default). Future versions may remove it.

fix

Remove 'headless' option from config. Use renderJavaScript to toggle headless browser usage.

affects: >=1.0.0

gotchaUsing stealth mode with concurrency >10 may trigger rate limiting on some sites due to rapid fingerprint rotation.

fix

Reduce concurrency or implement custom throttling via hooks.

affects: >=1.0.0

gotchaThe default export 'crawlex' returns an array; the return type is imported as CrawleXResult[].

fix

Use 'import type { CrawleXResult } from 'crawlex' for type annotations.

affects: >=1.0.0

breakingIn v1.0.0, the persistent queue was renamed from 'Queue' to 'CrawlQueue'. Imports using 'Queue' will break.

fix

Update imports to use 'CrawlQueue' instead of 'Queue'.

affects: >=1.0.0

Errors

Common errors & fixes

Error: Cannot find module 'crawlex' require() of ES Module

Using CommonJS require() to import CrawleX, which is ESM-only.

fix

Change to ESM imports: 'import crawlex from 'crawlex'' or use dynamic import() in CommonJS.

TypeError: crawlex is not a function

Importing named export instead of default export, or using wrong import syntax.

fix

Use default import: 'import crawlex from 'crawlex''.

Error [ERR_REQUIRE_ESM]: require() of ES Module .../crawlex/index.js from ... not supported

Trying to require() the package from a CommonJS context.

fix

Set 'type': 'module' in package.json or use import syntax in .mjs files.

Upgrade

Version history

1.0.4latest on npm

Audit

Dependencies

playwright-corerequiredUnderlying browser automation engine for headless rendering and rendering pool

gotrequiredHTTP client used for simple requests without rendering (fallback mode)

Agent activity

4 hits · last 30 days

node

Resources

githubhttps://github.com/forattini-dev/crawlex ↗

packagecrawlex ↗

homepagehttps://github.com/forattini-dev/crawlex ↗