OpenAI Token Counter

OpenAI Token Counter

Growing AI Skill PRO

Enter Calculation

Choose one of the following options:

GPT-6 models are not listed: OpenAI has not published their tokenizer in tiktoken yet.

Token Counting Method:

Formula

Tokens=len(encodeEncoding(Text))\text{Tokens} = \operatorname{len}\Big(\operatorname{encode}_{\,\text{Encoding}}\big(\text{Text}\big)\Big)
Show calculation steps (1)
  1. Tokens by OpenAI Model

    Tokens=len(encodeenc(Model)(Text))\text{Tokens} = \operatorname{len}\Big(\operatorname{encode}_{\,\text{enc}(\text{Model})}\big(\text{Text}\big)\Big)

    The model maps to its encoding, then Token count = number of tokens produced by encoding the input text.

Usage stats

Advertisement

Results

Token Count
9
Encoding: o200k_base · encode()
Token Statistics
Standard Token Count 9 (handles HTML tags specially)
Ordinary Token Count 9 (treats HTML as plain text)
Character Count 37
Word Count 7
Tokens per Word 1.29
Encoding Information
Encoding Type o200k_base
Counting Method encode()
Selected Via Encoding: o200k_base

Report an error on this page

This OpenAI Token Counter uses JTokkit v1.1.0, an open-source Java implementation of OpenAI's tiktoken.

Maven dependency: com.knuddels:jtokkit:1.1.0

JTokkit is developed and maintained by Knuddels GmbH and is available under the MIT license.

Implementation Details

This token counter is built using:

// Maven dependency
implementation 'com.knuddels:jtokkit:1.1.0'

// Token counting implementation
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.api.Encoding;
import com.knuddels.jtokkit.api.EncodingRegistry;
import com.knuddels.jtokkit.api.EncodingType;
import com.knuddels.jtokkit.api.ModelType;

// Usage example
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding encoding = registry.getEncoding(EncodingType.CL100K_BASE);
int tokenCount = encoding.countTokens(text);

JTokkit provides the same tokenization as OpenAI's official tiktoken library, ensuring accurate token counts.

OpenAI Token Counter Guide

What is tokenization?

Tokenization is the process of breaking text into smaller units called tokens. OpenAI models process text as tokens, not individual characters or words. Tokens can be as short as one character or as long as one word (e.g., "a" or "apple"). Common words are often a single token, while less common words might be split into multiple tokens.

The number of tokens is given by:

$$\text{Tokens} = \operatorname{len}\Big(\operatorname{encode}_{\,\text{Encoding}}\big(\text{Text}\big)\Big)$$

Important: Different models use different tokenizers, and your token count directly affects:

  • API costs (you pay per token for both input and output)
  • Context window usage (all models have a maximum token limit)
  • Processing speed (more tokens take longer to process)

Encoding Types

JTokkit provides several encoding types that match OpenAI's tokenizers:

Encoding Description Used By
cl100k_base Tokenizer of GPT-4, GPT-4 Turbo, GPT-3.5 Turbo and the text-embedding-3 / ada-002 embedding models GPT-4, GPT-4 Turbo, GPT-3.5 Turbo, text-embedding-3-small/large, text-embedding-ada-002
p50k_base Improved vocabulary with better handling of code GPT-3 davinci, curie, babbage, ada; Codex models
p50k_edit Specialized for edit models text-davinci-edit-001, code-davinci-edit-001
r50k_base Original GPT-3 tokenizer with basic vocabulary Older davinci, curie, babbage, ada models
o200k_base Current tokenizer of OpenAI's GPT-5, GPT-4.1, GPT-4o and o-series models (per OpenAI's tiktoken) GPT-5, GPT-4.1, GPT-4o, GPT-4o-mini, o1, o3, o4-mini

Models and Context Windows

Each OpenAI model has a maximum context window (number of tokens it can process in one request):

Model Max Tokens Encoding Type
gpt-5 272,000 o200k_base
gpt-4.1 1,047,576 o200k_base
gpt-4o 128,000 o200k_base
gpt-4o-mini 128,000 o200k_base
o3 200,000 o200k_base
o4-mini 200,000 o200k_base
o1 200,000 o200k_base
gpt-4-turbo 128,000 cl100k_base
gpt-4 8,192 cl100k_base
gpt-3.5-turbo 16,385 cl100k_base
text-embedding-3-large 8,191 cl100k_base
text-embedding-3-small 8,191 cl100k_base
text-embedding-ada-002 8,191 cl100k_base

Token Counting Methods

This calculator offers two counting methods:

Method Standard Ordinary
encode() Full tokenization that returns token IDs with special handling for HTML encodeOrdinary() treats HTML as plain text
countTokens() Optimized method that only counts tokens with special handling for HTML countTokensOrdinary() treats HTML as plain text

HTML handling: When your text contains HTML, standard methods recognize and process tags specially, while ordinary methods treat HTML tags as regular text. This can result in different token counts. Use ordinary methods if you want your HTML to be preserved exactly as written.

Token Counting Tips

  • Repetitive text often uses fewer tokens than unique text of the same length
  • Whitespace (spaces, tabs, newlines) counts as tokens
  • Special characters may be tokenized individually, increasing token count
  • Common words typically use fewer tokens than rare or technical terms
  • Non-English text often uses more tokens than equivalent English text
  • Code can be tokenized efficiently with the newer tokenizers (o200k_base, cl100k_base)

Common Token Count Examples

Text Tokens Note
"Hello world" 2 Common words often count as single tokens
"antidisestablishmentarianism" 5 Long words split into multiple tokens
"こんにちは" 3 Non-Latin characters often need more tokens
"1234567890" 3 Numbers are often chunked into multiple tokens
"<div>Hello</div>" 4-6 HTML tags may count differently between standard and ordinary methods
Diagram showing input text split into colored token chunks then counted
Text is split into tokens by the encoder, then the tokens are counted.

Interpreting Your Token Count

A token is the unit a model actually reads and writes: a whole word, part of a word, a punctuation mark, or a run of whitespace. The count shown is the length of the list produced when your text is encoded with the selected encoding, as in the formula \(\text{Tokens} = \operatorname{len}(\operatorname{encode}(\text{Text}))\).

Context window

Every model has a context window, which is the maximum number of tokens it can handle in one request. Input and output share that limit. The prompt you send and the reply the model generates both count toward it. If your prompt uses most of the window, little room is left for the response. Some APIs also let you set a separate cap on output length.

Billing

OpenAI's API bills per token, usually with separate rates for input and output tokens. Cost therefore scales roughly linearly with token count: doubling the tokens roughly doubles that part of the cost. Check OpenAI's current pricing page for actual rates.

Counts differ between encodings

Each model family uses its own encoding (tokenizer vocabulary). For example, GPT-4 and GPT-3.5-turbo use cl100k_base, while GPT-5, GPT-4.1, GPT-4o, GPT-4o-mini and the o-series (o1, o3, o4-mini) use o200k_base. Older models such as the davinci family use p50k_base or r50k_base. Because the vocabularies differ, the same text can produce different totals. Text in many non-English languages and some code often tokenizes into fewer tokens under o200k_base, while plain English often differs little. Always count with the encoding that matches the model you will call.

Chat-format overhead

This tool counts the raw text you paste. When you use a chat API, each message is wrapped with role and delimiter tokens, adding a few tokens per message, and the reply is also primed with a few extra tokens. Those are not included in a raw-text count, so the real prompt total for a multi-message conversation is slightly higher than the sum of the message texts. The exact overhead depends on the model and message format.

Advertisement

Token Counts Across Encodings

The samples below are illustrative. Counts were derived from the standard tiktoken vocabularies for these short inputs. Your own text, especially longer text, may behave differently, so use the tool above for exact counts.

Sample input Characters cl100k_base tokens o200k_base tokens Characters per token (cl100k_base)
Hello, world! 13 4 4 3.25
The quick brown fox jumps over the lazy dog. 44 10 10 4.4
1234567890 10 4 (123, 456, 789, 0) 4 (123, 456, 789, 0) 2.5

Short, common English text usually comes out near 4 characters per token, and it is the same under both encodings here. Digits are split into groups of up to three, so long numbers use many tokens per character.

Differences between encodings tend to show up more with non-English scripts, source code, and emoji. These can take several tokens per character, and o200k_base often needs fewer tokens for them than cl100k_base. Paste such text into the tool and switch the encoding to compare the totals directly.

Last updated:

Most popular in Miscellaneous

View all Miscellaneous calculators