Registry / networking / cobweb

cobweb

JSON →
library1.2.1rbrubygemsunverified

Cobweb is a web crawler that can use Resque to cluster crawls for large sites, with a statistics monitoring interface.

networking
gem install cobweb
github ↗homepage ↗docs ↗
INSTALL
IMPORT
SIG · COBWEB
C
cobweb
networkingrubyv1.2.1
harness data pending
Install & Compatibility
Where this runs

No compatibility data collected yet for this library.

Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

Cobweb
✓ require 'cobweb'

Start a crawl with a limit of 100 pages.

require 'cobweb' crawler = CobwebCrawler.new(:crawl_limit => 100) crawler.crawl('https://example.com')
Debug
Known issues

No known issues recorded.

Upgrade
Version history
1.2.1latest on RubyGems
Audit
Dependencies

No dependency data recorded yet.

Agent activity
29 hits · last 30 days
node
27
Amazon
1
OpenAI (training)
1
Resources
githubhttps://github.com/stewartmckee/cobweb ↗
homepagehttp://github.com/stewartmckee/cobweb ↗
docshttp://www.rubydoc.info/gems/cobweb/ ↗
cobweb — gem install cobweb · libregistry