The `robots-txt-parser` library provides a lightweight, promise-based solution for parsing `robots.txt` files efficiently in Node.js environments. It is currently at version 2.0.3, offering features such as comprehensive wildcard support in rules, configurable caching of `robots.txt` content, and flexible asynchronous operations via both promises and traditional callbacks. This package is specifically designed for developers building web crawlers, scrapers, and other automated bots that must adhere to website crawling policies. Key differentiators include its focus on Node.js, a clear API for determining URL crawlability, retrieving sitemaps, and managing crawl delays. The project maintains a stable release cadence, with the 2.x major version being actively supported since late 2018. Users can configure critical parameters such as the default user agent string and how the parser evaluates scenarios where allow/disallow rules are balanced.
npm install robots-txt-parserVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to initialize the robots.txt parser, fetch rules for a domain, and check URL crawlability using synchronous, promise-based, and callback methods. It also shows how to retrieve sitemaps.
Thoroughly test existing integrations when upgrading from 1.x. Review GitHub release notes for v2.0.0 if available for specific API changes.
Consistently use either the promise-based (`await robots.canCrawl(...)`) or callback-based approach. If using promises, always handle potential rejections with `try...catch` or `.catch()`.
Always ensure `await robots.useRobotsFor(domainUrl)` has successfully completed before making any `canCrawl` calls for URLs under `domainUrl`.
Carefully configure `userAgent` to match your bot's identity and understand the implications of `allowOnNeutral` for your crawling strategy. Test with various `robots.txt` files to ensure expected behavior.
Ensure you call the imported function: `const robots = robotsParser({ userAgent: 'MyBot' });`Use `await` keyword before promise-returning calls within an `async` function, or chain `.then()` and `.catch()` to handle the promise resolution and rejection.
Verify that `await robots.useRobotsFor(domainUrl)` completes successfully for the domain in question. Check your `userAgent` configuration to ensure it matches specific `User-agent` directives in the `robots.txt` file, or adjust `allowOnNeutral` if the default behavior is undesirable.
No dependency data recorded yet.