Registry / web-framework / express-robots-txt

express-robots-txt

library1.0.0jsnpmunverified

express-robots-txt is an Express.js middleware designed to either serve a static `robots.txt` file or dynamically generate one based on a JavaScript object or array configuration. The package is currently at version 1.0.0, with its last known publication approximately four years ago, indicating a stable but less actively developed state, likely in maintenance mode. It provides a straightforward API to define standard `robots.txt` directives such as `User-agent`, `Disallow`, `Allow`, `Crawl-delay`, `Sitemap` (supporting multiple entries), and `Host`. This flexibility allows developers to easily control how web crawlers interact with their Express applications, preventing unwanted indexing or specifying crawl behaviors, without needing to manage a static file manually for complex scenarios.

web-framework http-networking devops

npm install express-robots-txt

github ↗package ↗

INSTALL

IMPORT

SIG · EXPRESS-ROBOTS-TXT

express-robots-txt

web-frameworkjavascriptv1.0.0

Install

—

Import

—

Disk

—

Pass rate

0/ 6

Env Coverage0 / 6

glibc

18–22

musl

18–22

Install & Compatibility

Where this runs

tested against v? · npm install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

node 18–226 runs

build_error

glibc

node 18–226 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

robots

✓ import robots from 'express-robots-txt'

✗ import { robots } from 'express-robots-txt'

The primary export is a default export, typically imported without curly braces. The `robots` variable then becomes the middleware function.

robots (CommonJS)

✓ const robots = require('express-robots-txt')

For CommonJS environments, the module is imported directly into a variable, which then holds the middleware function.

This quickstart demonstrates how to set up `express-robots-txt` middleware to dynamically generate a `robots.txt` file with common directives or serve a static file.

import express from 'express'; import robots from 'express-robots-txt'; import path from 'path'; const app = express(); const port = process.env.PORT || 3000; // Option 1: Generate robots.txt from an object configuration app.use(robots({ UserAgent: '*', Disallow: ['/admin', '/private'], CrawlDelay: '10', Sitemap: [ 'https://www.example.com/sitemap.xml', 'https://www.example.com/sitemap-news.xml' ], Host: 'www.example.com' })); // Option 2 (alternative, comment out Option 1 to use): Serve a static robots.txt file // app.use(robots(path.join(__dirname, 'public/robots.txt'))); app.get('/', (req, res) => { res.send('Hello World! Check /robots.txt'); }); app.listen(port, () => { console.log(`Server listening on port ${port}`); console.log(`Open http://localhost:${port} and http://localhost:${port}/robots.txt`); });

Debug

Known issues

gotchaFor ES Module (ESM) usage, ensure your Node.js version supports `esm_conditional_exports` or higher. Older Node.js versions might not correctly resolve the `import` statement, defaulting to CommonJS behavior or throwing errors.

fix

Upgrade to a recent Node.js version (v12.17.0+ for stable ESM support) or use CommonJS `require()` syntax if ESM issues persist.

affects: <=1.0.0

gotchaThe `Host` directive in `robots.txt` is not universally supported by all web crawlers and is considered deprecated by Google. Relying solely on `Host` for canonicalization is not recommended.

fix

Use alternative methods for specifying canonical URLs, such as `<link rel="canonical" href="...">` in HTML headers, in addition to or instead of the `Host` directive.

affects: >=1.0.0

Errors

Common errors & fixes

TypeError: app.use() requires a middleware function but got a Object

The `robots` middleware was called without parentheses or a valid configuration object, thus not returning a function for Express to use.

fix

Ensure `app.use(robots(...))` is called with either a file path string or a configuration object/array, e.g., `app.use(robots({ UserAgent: '*', Disallow: '/' }))`.

My robots.txt output is empty or malformed.

The configuration object or array passed to `robots()` is invalid, or the specified file path does not exist/is inaccessible.

fix

Verify that the object structure follows the documented `robots.txt` directives (e.g., `UserAgent`, `Disallow`). If using a file, ensure the path is correct and the file is readable by the Node.js process.

Upgrade

Version history

1.0.0latest on npm

Audit

Dependencies

expressrequiredPeer dependency as it's an Express middleware.

Agent activity

9 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/modosc/express-robots-txt ↗

packagehttps://www.npmjs.com/package/express-robots-txt ↗