Registry / devops / fetch-filecache-for-crawling

fetch-filecache-for-crawling

JSON →
library5.1.1jsnpmunverified

Node.js module that wraps native fetch with a file-based HTTP cache for crawling purposes. Version 5.1.1 requires Node >= 18. It caches GET responses in a local folder (default `.cache`) and supports configurable cache expiration strategies (e.g., `once`, `never`, integer seconds). Unlike generic HTTP caches, it is designed for long-running crawls where cached files persist across application runs and conditional requests are sent. It is intentionally narrow in scope — not a general-purpose caching proxy but a specialized tool for crawling with opinionated defaults.

devops
github ↗package ↗homepage ↗
Install & Compatibility
Where this runs

No compatibility data collected yet for this library.

Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

ESM default export since v5 (native fetch). CommonJS require returns the function directly.

import fetch from 'fetch-filecache-for-crawling'

CJS require returns the default export directly; do not access .default.

const fetch = require('fetch-filecache-for-crawling')

Named import { fetch } is incorrect; use default import.

import fetch from 'fetch-filecache-for-crawling'

Demonstrates default import, caching with custom folder, and parallel fetching with logging.

import fetch from 'fetch-filecache-for-crawling'; const urls = [ 'https://caniuse.com/data.json', 'https://example.com' ]; Promise.all(urls.map(url => fetch(url, { logToConsole: true, cacheFolder: '.mycache' }) .then(response => response.text()) .then(text => console.log(`Fetched ${url}: ${text.length} chars`)) )).catch(err => console.error(err));
Debug
Known footguns
breakingv5 dropped direct node-fetch dependency and now wraps native fetch. This may break Node versions < 18.
deprecatedThe 'resetCache' option only works once per application lifecycle; subsequent calls ignore it.
gotchaCache key only uses URL; request headers are ignored. Two requests with same URL but different Accept headers will get same cached response.
gotchaThe 'refresh' parameter 'once' considers cache valid for entire application lifetime; does not refresh on restart.
gotchaCalling fetch.setParameter does not affect previously initiated requests; global parameters only apply to subsequent requests.
Upgrade
Version history

Breaking-change detection hasn't run for this library yet.

Audit
Security & dependencies

CVE tracking and dependency tree are planned for a later release.

Agent activity
9 hits · last 30 days
gptbot
3
ahrefsbot
1
Resources
githubhttps://github.com/tidoust/fetch-filecache-for-crawling ↗
packagefetch-filecache-for-crawling ↗
homepagehttps://github.com/tidoust/fetch-filecache-for-crawling#readme ↗