Registry / networking / robotex

robotex

JSON →
library1.0.0rbrubygemsunverified

A Ruby library for parsing and obeying robots.txt files to control web crawler access.

networkingutility
gem install robotex
github ↗homepage ↗docs ↗
INSTALL
IMPORT
SIG · ROBOTEX
R
robotex
networkingrubyv1.0.0
harness data pending
Install & Compatibility
Where this runs

No compatibility data collected yet for this library.

Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

Robotex
✓ require 'robotex'

Check if a URL is allowed by robots.txt

require 'robotex' robotex = Robotex.new puts robotex.allowed?('https://example.com/')
Debug
Known issues

No known issues recorded.

Upgrade
Version history
1.0.0latest on RubyGems
Audit
Dependencies

No dependency data recorded yet.

Agent activity
7 hits · last 30 days
node
6
OpenAI (training)
1
Resources
githubhttps://github.com/chriskite/robotex ↗
homepagehttp://www.github.com/chriskite/robotex ↗
docshttp://www.rubydoc.info/gems/robotex/ ↗
robotex — gem install robotex · libregistry