Registry / networking / temoto-robotstxt

temoto-robotstxt

JSON →

library1.1.2gogounverified

Package robotstxt implements the robots.txt Exclusion Protocol with various extensions.

networking

go get github.com/temoto/robotstxt

github ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · TEMOTO-ROBOTSTXT

temoto-robotstxt

networkinggov1.1.2

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

robotstxt

✓ github.com/temoto/robotstxt

Parse and test robots.txt rules.

package main import ( "fmt" "net/http" "github.com/temoto/robotstxt" ) func main() { resp, _ := http.Get("https://example.com/robots.txt") defer resp.Body.Close() data, _ := robotstxt.FromResponse(resp) fmt.Println(data.TestAgent("/", "MyBot")) }

Debug

Known issues

No known issues recorded.

Upgrade

Version history

1.1.2latest on pkg.go.dev

Audit

Dependencies

No dependency data recorded yet.

Agent activity

13 hits · last 30 days

node

Amazon

OpenAI (training)

Resources

githubhttps://github.com/temoto/robotstxt ↗

homepagehttps://github.com/temoto/robotstxt ↗

docshttps://pkg.go.dev/github.com/temoto/robotstxt#section-documentation ↗