Registry / http-networking / robot-directives

robot-directives

JSON →
library0.4.0jsnpmunverified

The `robot-directives` package (current stable version 0.4.0) provides a focused utility for parsing and interpreting robot directives found within HTML `<meta name="robots">` tags and `X-Robots-Tag` HTTP headers. It allows developers to programmatically determine a crawler's allowed or disallowed actions based on these instructions, such as `noindex`, `nofollow`, `noarchive`, and `unavailable_after`. The library handles the cascading logic of multiple directives, user-agent specific rules, and resolves conflicts based on a `restrictive` default (mimicking Googlebot's behavior). It explicitly differentiates itself by not handling the underlying HTML parsing, requiring users to extract meta tag content themselves. While not on a rapid release cycle, the package offers a stable API for its specialized parsing tasks, including a comprehensive set of constants for all standard robot directives and static methods for general utility.

http-networkingweb-framework
npm install robot-directives
github ↗package ↗
INSTALL
IMPORT
SIG · ROBOT-DIRECTIVES
R
robot-directives
http-networkingjavascriptv0.4.0
Install
—
Import
—
Disk
—
Pass rate
0/ 6
Env Coverage0 / 6
glibc
18–22
musl
18–22
Install & Compatibility
Where this runs
tested against v? · npm install
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
node 18–226 runs
build_error
glibc
node 18–226 runs
build_error
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

RobotDirectives
✓ const RobotDirectives = require('robot-directives');
✗ import { RobotDirectives } from 'robot-directives';
This package is primarily a CommonJS module. For ESM environments, use `import RobotDirectives from 'robot-directives';` for the default export.
RobotDirectives.NOINDEX
✓ const { NOINDEX } = RobotDirectives;
✗ import { NOINDEX } from 'robot-directives';
Directive constants are static properties of the `RobotDirectives` class and must be accessed via the class constructor.
RobotDirectives.isBot
✓ const { isBot } = RobotDirectives;
✗ import { isBot } from 'robot-directives';
The `isBot` function is a static method of the `RobotDirectives` class, not a named export.

Demonstrates how to instantiate `RobotDirectives`, add directives from HTTP headers and HTML meta tags, and then query their state using `is()` with specific user agents and time overrides. It also shows the usage of the static `isBot()` helper.

const RobotDirectives = require('robot-directives'); // Instantiate with default options const robots = new RobotDirectives({ // Example: Override default userAgent if needed // userAgent: 'Googlebot/2.1 (web crawler) (+http://www.google.com/bot.html)', // Example: Override current time for 'unavailable_after' testing // currentTime: () => new Date('jan 1 2025').getTime() }); // Add directives from an X-Robots-Tag HTTP header robots.header('googlebot: noindex, nosnippet'); // Add directives from HTML meta tags robots.meta('robots', 'noarchive,nofollow'); robots.meta('bingbot', 'unavailable_after: 1-Jan-3000 00:00:00 EST'); // Check specific directives console.log('Is nofollow?', robots.is(RobotDirectives.NOFOLLOW)); // Expected: true console.log('Is noindex for Googlebot?', robots.is(RobotDirectives.NOINDEX, { userAgent: 'Googlebot' })); // Expected: true console.log('Is noarchive?', robots.is(RobotDirectives.NOARCHIVE)); // Expected: true // Check for a directive that is not present console.log('Is index?', robots.is(RobotDirectives.INDEX)); // Expected: false // Check if 'unavailable_after' has passed (example: assuming current time is after 3000) console.log('Is noindex for Bingbot after 3000?', robots.is(RobotDirectives.NOINDEX, { currentTime: () => new Date('Jan 2 3000').getTime(), // Set current time past the unavailable_after date userAgent: 'Bingbot/2.0' })); // Expected: true // Use static helper function console.log('Is "googlebot" a recognized bot name?', RobotDirectives.isBot('googlebot')); // Expected: true console.log('Is "mycustombot" a recognized bot name?', RobotDirectives.isBot('mycustombot')); // Expected: false
Debug
Known issues
gotchaThe `allIsReadonly` option defaults to `true`. This means declaring an `'all'` directive will not overwrite other directives, which might be counter-intuitive if expecting 'all' to be absolute. Most search crawlers behave this way, but it's important to be aware of.
fix
If you require 'all' to be an absolute override, set `allIsReadonly: false` in the constructor options. Be aware this deviates from common crawler behavior.
affects: >=0.1.0
gotchaThe `restrictive` option defaults to `true`, resolving directive conflicts (e.g., `noindex,index`) by selecting the most restrictive value (`noindex`). While this mimics Googlebot, other crawlers might resolve conflicts differently.
fix
If your target crawler doesn't follow restrictive conflict resolution, set `restrictive: false` in the constructor options. You may then need to implement custom conflict resolution logic.
affects: >=0.1.0
gotchaThe `unavailable_after` directive's evaluation depends on the `currentTime` option. If `currentTime` is not correctly configured (e.g., time zone issues, static date), the `unavailable_after` directive might not be interpreted as expired when it should be, or vice-versa.
fix
Ensure `currentTime` in the constructor options or `is()` method options accurately reflects the desired current time, especially when dealing with dates and time zones.
affects: >=0.1.0
gotchaThis library explicitly states it is NOT responsible for parsing HTML. You must manually extract the `content` attribute from `<meta name="robots">` tags and pass it to the `meta()` method.
fix
Use a dedicated HTML parser (e.g., `cheerio`, `jsdom`) to extract the meta tag content before passing it to `robot-directives`.
affects: >=0.1.0
Errors
Common errors & fixes
TypeError: RobotDirectives is not a constructor
Attempting to use `import { RobotDirectives } from 'robot-directives';` or other incorrect destructuring for a CommonJS default export.
fix
For CommonJS, use `const RobotDirectives = require('robot-directives');`. For ESM, use `import RobotDirectives from 'robot-directives';`.
ReferenceError: require is not defined
Trying to use `require()` in an ES Module context without proper setup or bundler configuration.
fix
If in an ESM file (e.g., `type: module` in `package.json`), use `import RobotDirectives from 'robot-directives';` instead of `const RobotDirectives = require('robot-directives');`.
Directive 'X' not working as expected (e.g., 'all' isn't overriding other directives)
Misunderstanding the default behavior of `allIsReadonly` or `restrictive` options.
fix
Review the constructor options for `allIsReadonly` (default `true`) and `restrictive` (default `true`). Adjust them in the `new RobotDirectives(options)` call if the default behavior doesn't match your expectations.
Upgrade
Version history
0.4.0latest on npm
Audit
Dependencies

No dependency data recorded yet.

Agent activity
14 hits · last 30 days
node
12
OpenAI (training)
1
Resources
githubhttps://github.com/stevenvachon/robot-directives ↗
packagehttps://www.npmjs.com/package/robot-directives ↗
robot-directives — npm install robot-directives · libregistry