Summary
Turns 'how long is this prompt' into a number you can act on.
Counting is the part people guess at; this one counts with the real encoder and then converts the count into money for a specific model, so a decision about trimming a prompt has an actual figure behind it.
What it is
A wrapper around OpenAI's tiktoken library: a token count for any text, and a cost estimate for that text against a named model.
What you get
- A token count for any text you pass in
- A choice of encodings — cl100k_base by default, with p50k_base, p50k_edit and r50k_base available
- The first token IDs alongside the count, so you can see how the text was split
- An input cost, an output cost and a combined figure for a named model
- Model coverage spanning gpt-4, gpt-4-turbo and the claude-3 family
Requirements
Node.js 16 or higher, plus Python 3.6 or higher and pip on the same machine — the tiktoken Python library is installed on first run.