Registry / devops / fetch-filecache-for-crawling

fetch-filecache-for-crawling

library5.1.1jsnpmunverified

Node.js module that wraps native fetch with a file-based HTTP cache for crawling purposes. Version 5.1.1 requires Node >= 18. It caches GET responses in a local folder (default `.cache`) and supports configurable cache expiration strategies (e.g., `once`, `never`, integer seconds). Unlike generic HTTP caches, it is designed for long-running crawls where cached files persist across application runs and conditional requests are sent. It is intentionally narrow in scope — not a general-purpose caching proxy but a specialized tool for crawling with opinionated defaults.

devops

npm install fetch-filecache-for-crawling

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · FETCH-FILECACHE-FO

fetch-filecache-for-crawling

devopsjavascriptv5.1.1

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

default

✓ import fetch from 'fetch-filecache-for-crawling'

✗ const { fetch } = require('fetch-filecache-for-crawling')

ESM default export since v5 (native fetch). CommonJS require returns the function directly.

default

✓ const fetch = require('fetch-filecache-for-crawling')

✗ const fetch = require('fetch-filecache-for-crawling').default

CJS require returns the default export directly; do not access .default.

default

✓ import fetch from 'fetch-filecache-for-crawling'

✗ import { fetch } from 'fetch-filecache-for-crawling'

Named import { fetch } is incorrect; use default import.

Demonstrates default import, caching with custom folder, and parallel fetching with logging.

import fetch from 'fetch-filecache-for-crawling'; const urls = [ 'https://caniuse.com/data.json', 'https://example.com' ]; Promise.all(urls.map(url => fetch(url, { logToConsole: true, cacheFolder: '.mycache' }) .then(response => response.text()) .then(text => console.log(`Fetched ${url}: ${text.length} chars`)) )).catch(err => console.error(err));

Debug

Known issues

breakingv5 dropped direct node-fetch dependency and now wraps native fetch. This may break Node versions < 18.

fix

Upgrade Node to >=18 or pin to v4.x

affects: >=5.0.0

deprecatedThe 'resetCache' option only works once per application lifecycle; subsequent calls ignore it.

fix

Manually delete cache folder if you need repeated resets

affects: >=1.0.0

gotchaCache key only uses URL; request headers are ignored. Two requests with same URL but different Accept headers will get same cached response.

fix

Consider this limitation or wait for #3 fix

affects: >=1.0.0 <6.0.0

gotchaThe 'refresh' parameter 'once' considers cache valid for entire application lifetime; does not refresh on restart.

fix

Use 'default' or integer seconds for time-based refresh

affects: >=1.0.0

gotchaCalling fetch.setParameter does not affect previously initiated requests; global parameters only apply to subsequent requests.

fix

Set parameters before any fetch calls

affects: >=1.0.0

Errors

Common errors & fixes

TypeError: fetch is not a function

Using named import instead of default import in ESM

fix

Use `import fetch from 'fetch-filecache-for-crawling'`

Error: The fetch implementation only supports GET requests

Attempted a POST or other HTTP method

fix

Use only GET requests or wait for HTTP method support

Error: Could not find module 'fetch-filecache-for-crawling'

Package not installed or Node.js version < 18

fix

Run npm install fetch-filecache-for-crawling and ensure Node >= 18

Upgrade

Version history

5.1.1latest on npm

Audit

Dependencies

node-fetchoptionalUsed as underlying fetch implementation in older versions; v5 uses native fetch

Agent activity

2 hits · last 30 days

node

Resources

githubhttps://github.com/tidoust/fetch-filecache-for-crawling ↗

packagefetch-filecache-for-crawling ↗

homepagehttps://github.com/tidoust/fetch-filecache-for-crawling#readme ↗