`robots-parser` is a JavaScript/TypeScript library designed to parse `robots.txt` files according to the draft specification. As of version 3.0.1, it provides robust support for directives such as `User-agent`, `Allow`, `Disallow`, `Sitemap`, `Crawl-delay`, and `Host`, including advanced features like wildcard (`*`) and end-of-line (`$`) matching for paths. The library maintains an active development status, with recent releases addressing critical bug fixes (e.g., HTTPS URL port handling in 3.0.1) and improving compatibility (e.g., using global URL object in 3.0.0, adding TypeScript definitions in 2.4.0). Its primary differentiator is its adherence to the specification and comprehensive feature set for accurately determining URL crawlability for different user agents, making it a reliable choice for web crawlers and SEO tools.
Install & Compatibility
Where this runs
tested against v? · npm install
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslnode 18–226 runs
build_error
glibcnode 18–226 runs
build_error
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
The primary parsing function is a default export in ESM. For CommonJS, it's the module.exports.
import robotsParser from 'robots-parser';
CommonJS `require` syntax for Node.js environments. Node.js v10+ is required since v3.0.0.
const robotsParser = require('robots-parser');
TypeScript type for the instantiated robots parser object. Types were added in v2.4.0.
import type { RobotsParser } from 'robots-parser';
Demonstrates parsing a `robots.txt` string, checking URL crawlability for different user-agents, retrieving sitemaps, crawl delays, and preferred host from the parsed rules. It also illustrates how to handle out-of-scope URLs.
import robotsParser from 'robots-parser';
async function runRobotsParserExample() {
const baseUrl = 'http://www.example.com';
const robotsTxtContent = `
User-agent: *
Disallow: /dir/
Disallow: /test.html
Allow: /dir/test.html
Allow: /test.html
Crawl-delay: 1
Sitemap: ${baseUrl}/sitemap.xml
Host: example.com
`.trim();
// In a real application, you'd typically fetch this content from a URL:
// const response = await fetch(`${baseUrl}/robots.txt`);
// const fetchedRobotsTxtContent = await response.text();
const robots = robotsParser(`${baseUrl}/robots.txt`, robotsTxtContent);
console.log('Is http://www.example.com/test.html allowed for Sams-Bot/1.0?', robots.isAllowed('http://www.example.com/test.html', 'Sams-Bot/1.0'));
console.log('Is http://www.example.com/dir/test.html allowed for Sams-Bot/1.0?', robots.isAllowed('http://www.example.com/dir/test.html', 'Sams-Bot/1.0'));
console.log('Is http://www.example.com/dir/test2.html disallowed for Sams-Bot/1.0?', robots.isDisallowed('http://www.example.com/dir/test2.html', 'Sams-Bot/1.0'));
console.log('Crawl delay for Sams-Bot/1.0:', robots.getCrawlDelay('Sams-Bot/1.0'));
console.log('Sitemaps:', robots.getSitemaps());
console.log('Preferred Host:', robots.getPreferredHost());
// Demonstrating undefined for invalid URLs (i.e., not matching the base URL)
console.log('Is an out-of-scope URL allowed?', robots.isAllowed('http://www.anotherdomain.com/path', 'Sams-Bot/1.0'));
}
runRobotsParserExample();
Upgrade
Version history
Breaking-change detection hasn't run for this library yet.
Audit
Security & dependencies
CVE tracking and dependency tree are planned for a later release.