tiktoken-ruby

library0.0.16rbrubygemsunverified

An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer used by OpenAI, for counting tokens in text before sending to OpenAI APIs.

gem install tiktoken_ruby

INSTALL

IMPORT

SIG · TIKTOKEN-RUBY

tiktoken-ruby

utilityrubyv0.0.16

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Tiktoken

✓ require 'tiktoken_ruby'

Count tokens for a string using GPT-4 encoding

require 'tiktoken_ruby' enc = Tiktoken.encoding_for_model('gpt-4') tokens = enc.encode('Hello, world!') puts tokens.length

Debug

Known issues

gotchaNot an official OpenAI gem; may lag behind API changes

fix

Check for updates or use official OpenAI gem if available

affects: >= 0.0.0

Upgrade

Version history

0.0.16latest on RubyGems

Audit

Dependencies

No dependency data recorded yet.

Agent activity

9 hits · last 30 days

node

Resources