Registry / utility / pkoukk-tiktoken-go

pkoukk-tiktoken-go

JSON →

library0.1.8gogounverified

Go port of OpenAI's tiktoken tokenizer for encoding and decoding text tokens.

utility

go get github.com/pkoukk/tiktoken-go

github ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · PKOUKK-TIKTOKEN-GO

pkoukk-tiktoken-go

utilitygov0.1.8

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

tiktoken

✓ github.com/pkoukk/tiktoken-go

Encode a string using the cl100k_base encoding.

package main import ( "fmt" "github.com/pkoukk/tiktoken-go" ) func main() { tkm, err := tiktoken.GetEncoding("cl100k_base") if err != nil { panic(err) } encoded := tkm.Encode("hello world", nil, nil) fmt.Println(encoded) }

Debug

Known issues

No known issues recorded.

Upgrade

Version history

0.1.8latest on pkg.go.dev

Audit

Dependencies

No dependency data recorded yet.

Agent activity

5 hits · last 30 days

node

Amazon

Resources

githubhttps://github.com/pkoukk/tiktoken-go ↗

homepagehttps://github.com/pkoukk/tiktoken-go ↗

docshttps://pkg.go.dev/github.com/pkoukk/tiktoken-go#section-documentation ↗