robotstxt

library0.5.4rbrubygemsunverified

Robotstxt Parser allows you to check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap rules.

gem install robotstxt

INSTALL

IMPORT

SIG · ROBOTSTXT

robotstxt

networkingrubyv0.5.4

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Robotstxt

✓ require 'robotstxt'

Check if a URL is allowed by robots.txt rules.

require 'robotstxt' parser = Robotstxt::Parser.new('User-agent: *\nDisallow: /admin') parser.allowed?('https://example.com/admin')

Debug

Known issues

No known issues recorded.

Upgrade

Version history

0.5.4latest on RubyGems

Audit

Dependencies

No dependency data recorded yet.

Agent activity

9 hits · last 30 days

node

OpenAI (training)

Resources