Registry / networking / robots

robots

JSON →
library0.10.1rbrubygemsunverified

Parses robots.txt files to determine allowed and disallowed paths for web crawlers.

networkingutility
gem install robots
github ↗homepage ↗docs ↗
INSTALL
IMPORT
SIG · ROBOTS
R
robots
networkingrubyv0.10.1
harness data pending
Install & Compatibility
Where this runs

No compatibility data collected yet for this library.

Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

Robots
✓ require 'robots'

Check if a path is allowed by the robots.txt file.

require 'robots' robots = Robots.new('http://example.com/robots.txt') puts robots.allowed?('/admin')
Debug
Known issues

No known issues recorded.

Upgrade
Version history
0.10.1latest on RubyGems
Audit
Dependencies

No dependency data recorded yet.

Agent activity
25 hits · last 30 days
node
22
OpenAI (training)
1
Resources
githubhttps://github.com/fizx/robots ↗
homepagehttp://github.com/fizx/robots ↗
docshttp://www.rubydoc.info/gems/robots/ ↗
robots — gem install robots · libregistry