Registry / devops / gatsby-source-filesystem-with-queue

gatsby-source-filesystem-with-queue

library5.3.1jsnpmunverified

A Gatsby source plugin for sourcing local files into a Gatsby application, identical in usage to gatsby-source-filesystem but with an asynchronous queue that limits concurrent file reads to prevent EMFILE errors on large sites. Version 5.3.1 requires Node >=18 and Gatsby ^5.0.0-next. Key differentiator: configurable concurrent file limit (default 800) via GATSBY_CONCURRENT_FILES environment variable, and concurrent download limit (default 200) via GATSBY_CONCURRENT_DOWNLOAD. Ships TypeScript types. Maintains full compatibility with transformer plugins like gatsby-transformer-remark and gatsby-transformer-json. Only difference from upstream is the queuing mechanism; all options, helper functions (createFilePath, createRemoteFileNode, createFileNodeFromBuffer), and query patterns are identical.

devops storage

npm install gatsby-source-filesystem-with-queue

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · GATSBY-SOURCE-FILE

gatsby-source-filesystem-with-queue

devopsjavascriptv5.3.1

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

createFilePath

✓ import { createFilePath } from 'gatsby-source-filesystem-with-queue'

✗ const createFilePath = require('gatsby-source-filesystem-with-queue').createFilePath

ESM-only. Named export, not default. Don't use destructuring from default import.

createRemoteFileNode

✓ import { createRemoteFileNode } from 'gatsby-source-filesystem-with-queue'

✗ import createRemoteFileNode from 'gatsby-source-filesystem-with-queue'

Named export. Mistaking it for default export is common.

createFileNodeFromBuffer

✓ import { createFileNodeFromBuffer } from 'gatsby-source-filesystem-with-queue'

✗ import { createFileNodeFromBuffer } from '../node_modules/gatsby-source-filesystem-with-queue'

Relative path import is unnecessary and breaks. Import from package name only.

Configure plugin in gatsby-config.js with a source directory and query the created File nodes in a page component.

// gatsby-config.js module.exports = { plugins: [ { resolve: `gatsby-source-filesystem-with-queue`, options: { name: `pages`, path: `${__dirname}/src/pages/`, }, }, ], } // Query in Gatsby page/component import { graphql } from 'gatsby' export const query = graphql` { allFile(filter: { sourceInstanceName: { eq: "pages" } }) { edges { node { extension dir modifiedTime } } } } `

Debug

Known issues

breakingMinimum Node version is 18.0.0 — using older versions will fail with cryptic errors.

fix

Update Node to >=18.0.0, or install an older version of the package (e.g., 4.x).

affects: >=5.3.1

breakingPeer dependency on Gatsby ^5.0.0-next — incompatible with Gatsby 4.x or earlier.

fix

Ensure your project uses Gatsby 5.x. If on Gatsby 4, use gatsby-source-filesystem instead.

affects: >=5.0.0

deprecatedThe package is a fork of gatsby-source-filesystem with identical API — if you don't need the queue, consider using the original to avoid maintenance risk.

fix

Switch to gatsby-source-filesystem unless you specifically need queue concurrency control.

affects: >=5.3.1

gotchaEnvironment variable GATSBY_CONCURRENT_FILES default is 800, not 8000 as older docs might suggest — lowering it may help EMFILE errors but slow down builds.

fix

Set GATSBY_CONCURRENT_FILES to a lower value (e.g., 200) if you experience 'EMFILE: too many open files' errors.

affects: >=5.3.1

gotchaEnvironment variable GATSBY_CONCURRENT_DOWNLOAD controls remote file node downloads (default 200) — setting too high may exhaust system resources.

fix

Keep GATSBY_CONCURRENT_DOWNLOAD reasonable (e.g., <=500) or adjust based on system limits.

affects: >=5.3.1

gotchaignore option globs are merged with defaults — you cannot override defaults, only add to them.

fix

If you need to remove a default ignore, you'll need to fork the plugin. Alternatively, use gatsby-source-filesystem which may allow customization.

affects: >=5.3.1

deprecatedcreateFilePath's node and getNode parameters are required — omitting them causes the query to silently fail.

fix

Always pass node (File node or transformer-created node with parent) and getNode function from onCreateNode API.

affects: >=5.3.1

Errors

Common errors & fixes

EMFILE: too many open files

Simultaneous file reads exceed OS limit (often around 256 on macOS/Linux).

fix

Set environment variable GATSBY_CONCURRENT_FILES to a lower value, e.g., 100, or increase the OS file limit using `ulimit -n 2048`.

Cannot find module 'gatsby-source-filesystem-with-queue'

Package not installed or not in node_modules.

fix

Run `npm install gatsby-source-filesystem-with-queue` or `yarn add gatsby-source-filesystem-with-queue`.

The "path" argument must be of type string. Received undefined

Missing or undefined 'path' option in plugin configuration.

fix

Ensure the 'path' option is a string, e.g., path: `${__dirname}/src/pages/`.

TypeError: Cannot read properties of undefined (reading 'createFilePath')

Attempting to call createFilePath without importing it, or importing as default instead of named.

fix

Import as named export: `import { createFilePath } from 'gatsby-source-filesystem-with-queue'`.

Upgrade

Version history

5.3.1latest on npm

Audit

Dependencies

gatsbyrequiredpeer dependency - requires Gatsby ^5.0.0-next to run

Agent activity

29 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/FraserThompson/gatsby-source-filesystem-with-queue ↗

packagegatsby-source-filesystem-with-queue ↗

homepagehttps://github.com/FraserThompson/gatsby-source-filesystem-with-queue ↗