Registry / http-networking / robots-parser

robots-parser

library3.0.1jsnpmunverified

`robots-parser` is a JavaScript/TypeScript library designed to parse `robots.txt` files according to the draft specification. As of version 3.0.1, it provides robust support for directives such as `User-agent`, `Allow`, `Disallow`, `Sitemap`, `Crawl-delay`, and `Host`, including advanced features like wildcard (`*`) and end-of-line (`$`) matching for paths. The library maintains an active development status, with recent releases addressing critical bug fixes (e.g., HTTPS URL port handling in 3.0.1) and improving compatibility (e.g., using global URL object in 3.0.0, adding TypeScript definitions in 2.4.0). Its primary differentiator is its adherence to the specification and comprehensive feature set for accurately determining URL crawlability for different user agents, making it a reliable choice for web crawlers and SEO tools.

http-networking

npm install robots-parser

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · ROBOTS-PARSER

robots-parser

http-networkingjavascriptv3.0.1

Install

—

Import

—

Disk

—

Pass rate

0/ 6

Env Coverage0 / 6

glibc

18–22

musl

18–22

Install & Compatibility

Where this runs

tested against v? · npm install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

node 18–226 runs

build_error

glibc

node 18–226 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

robotsParser

✓ import robotsParser from 'robots-parser';

✗ import { robotsParser } from 'robots-parser';

The primary parsing function is a default export in ESM. For CommonJS, it's the module.exports.

robotsParser

✓ const robotsParser = require('robots-parser');

✗ import robotsParser from 'robots-parser';

CommonJS `require` syntax for Node.js environments. Node.js v10+ is required since v3.0.0.

RobotsParser

✓ import type { RobotsParser } from 'robots-parser';

✗ import { RobotsParser } from 'robots-parser';

TypeScript type for the instantiated robots parser object. Types were added in v2.4.0.

Demonstrates parsing a `robots.txt` string, checking URL crawlability for different user-agents, retrieving sitemaps, crawl delays, and preferred host from the parsed rules. It also illustrates how to handle out-of-scope URLs.

import robotsParser from 'robots-parser'; async function runRobotsParserExample() { const baseUrl = 'http://www.example.com'; const robotsTxtContent = ` User-agent: * Disallow: /dir/ Disallow: /test.html Allow: /dir/test.html Allow: /test.html Crawl-delay: 1 Sitemap: ${baseUrl}/sitemap.xml Host: example.com `.trim(); // In a real application, you'd typically fetch this content from a URL: // const response = await fetch(`${baseUrl}/robots.txt`); // const fetchedRobotsTxtContent = await response.text(); const robots = robotsParser(`${baseUrl}/robots.txt`, robotsTxtContent); console.log('Is http://www.example.com/test.html allowed for Sams-Bot/1.0?', robots.isAllowed('http://www.example.com/test.html', 'Sams-Bot/1.0')); console.log('Is http://www.example.com/dir/test.html allowed for Sams-Bot/1.0?', robots.isAllowed('http://www.example.com/dir/test.html', 'Sams-Bot/1.0')); console.log('Is http://www.example.com/dir/test2.html disallowed for Sams-Bot/1.0?', robots.isDisallowed('http://www.example.com/dir/test2.html', 'Sams-Bot/1.0')); console.log('Crawl delay for Sams-Bot/1.0:', robots.getCrawlDelay('Sams-Bot/1.0')); console.log('Sitemaps:', robots.getSitemaps()); console.log('Preferred Host:', robots.getPreferredHost()); // Demonstrating undefined for invalid URLs (i.e., not matching the base URL) console.log('Is an out-of-scope URL allowed?', robots.isAllowed('http://www.anotherdomain.com/path', 'Sams-Bot/1.0')); } runRobotsParserExample();

Debug

Known issues

breakingVersion 3.0.0 introduced a breaking change by relying on the global `URL` object, removing support for Node.js versions prior to 10. Users on older Node.js environments must upgrade their Node.js version or remain on `robots-parser` v2.x.

fix

Upgrade Node.js to version 10 or higher, or downgrade `robots-parser` to a 2.x release.

affects: >=3.0.0

gotchaThe `isAllowed()` and `isDisallowed()` methods can return `undefined` if the provided URL is not considered valid for the given `robots.txt`'s base URL, rather than a strict boolean. Developers should explicitly check for `undefined` in their logic.

fix

Handle `undefined` return values explicitly: `if (result === true) { ... } else if (result === false) { ... } else { /* URL out of scope or invalid */ }`

affects: >=1.0.0

breakingPrior to version 3.0.1, `https:` URLs without an explicit port would incorrectly default to port `80` instead of `443` during comparison. This could lead to incorrect `isAllowed` or `isDisallowed` results when comparing `https://example.com/` with `https://example.com:443/`.

fix

Upgrade to version `3.0.1` or higher to resolve the `https` port default issue.

affects: <3.0.1

gotchaIn versions prior to 2.3.0, passing 'constructor' as a user-agent string to `isAllowed()` or `isDisallowed()` could lead to an internal error due to a bug. This edge case was fixed in v2.3.0.

fix

Upgrade to version `2.3.0` or higher to prevent potential errors with specific user-agent strings.

affects: <2.3.0

gotchaThe `getMatchingLineNumber` method can return `-1` if no matching directive is found, or `undefined` if a rule was manually added without an associated line number. It does not consistently return a single type for 'no match' scenarios.

fix

Check for both `undefined` and `-1` when evaluating the result of `getMatchingLineNumber`.

affects: >=1.0.0

Errors

Common errors & fixes

TypeError: URL is not a constructor

Running `robots-parser` v3.0.0+ on Node.js versions older than 10.0.0, which lack the global `URL` object required by the library.

fix

Upgrade your Node.js environment to version 10.0.0 or higher. Alternatively, downgrade `robots-parser` to a 2.x release if Node.js upgrade is not an option.

TypeError: robots.isAllowed is not a function

The `robotsParser` function was called with invalid arguments (e.g., incorrect URL format, malformed `robots.txt` content), preventing proper instantiation and returning an invalid object or `undefined`.

fix

Ensure `robotsParser` is called with a valid base URL for the `robots.txt` file and correctly formatted `robots.txt` content (as a string). Verify the content for syntax errors or unexpected structures.

Incorrect `isAllowed` or `isDisallowed` result for HTTPS URLs

Using `robots-parser` versions prior to 3.0.1, leading to `https:` URLs without explicit ports being incorrectly treated as port `80` instead of `443` during internal comparisons.

fix

Update `robots-parser` to version `3.0.1` or newer to correctly handle `https` URL port comparisons and ensure accurate permission checks.

Upgrade

Version history

3.0.1latest on npm

Audit

Dependencies

No dependency data recorded yet.

Agent activity

18 hits · last 30 days

node

Resources

githubhttps://github.com/samclarke/robots-parser ↗

packagehttps://www.npmjs.com/package/robots-parser ↗

homepagehttps://www.npmjs.com/package/robots-parser ↗