Registry / utility / cohere-ai-tokenizer

cohere-ai-tokenizer

library1.1.2gogounverified

A Go tokenizer for Cohere's language models, converting text into tokens for API consumption.

go get github.com/cohere-ai/tokenizer

INSTALL

IMPORT

SIG · COHERE-AI-TOKENIZE

cohere-ai-tokenizer

utilitygov1.1.2

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

tokenizer

✓ github.com/cohere-ai/tokenizer

Tokenizes a string into Cohere tokens.

package main import ( "fmt" "github.com/cohere-ai/tokenizer" ) func main() { tokens := tokenizer.Tokenize("Hello, world!") fmt.Println(tokens) }

Debug

Known issues

No known issues recorded.

Upgrade

Version history

1.1.2latest on pkg.go.dev

Audit

Dependencies

No dependency data recorded yet.

Agent activity

5 hits · last 30 days

node

Amazon

Resources