This OpenAI Token Counter uses JTokkit v1.1.0, an open-source Java implementation of OpenAI's tiktoken.
Maven dependency: com.knuddels:jtokkit:1.1.0
JTokkit is developed and maintained by Knuddels GmbH and is available under the MIT license.
Implementation Details
This token counter is built using:
// Maven dependency
implementation 'com.knuddels:jtokkit:1.1.0'
// Token counting implementation
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.api.Encoding;
import com.knuddels.jtokkit.api.EncodingRegistry;
import com.knuddels.jtokkit.api.EncodingType;
import com.knuddels.jtokkit.api.ModelType;
// Usage example
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding encoding = registry.getEncoding(EncodingType.CL100K_BASE);
int tokenCount = encoding.countTokens(text);
JTokkit provides the same tokenization as OpenAI's official tiktoken library, ensuring accurate token counts.
OpenAI Token Counter Guide
What is tokenization?
Tokenization is the process of breaking text into smaller units called tokens. OpenAI models process text as tokens, not individual characters or words. Tokens can be as short as one character or as long as one word (e.g., "a" or "apple"). Common words are often a single token, while less common words might be split into multiple tokens.
The number of tokens is given by:
$$\text{Tokens} = \operatorname{len}\Big(\operatorname{encode}_{\,\text{Encoding}}\big(\text{Text}\big)\Big)$$Important: Different models use different tokenizers, and your token count directly affects:
- API costs (you pay per token for both input and output)
- Context window usage (all models have a maximum token limit)
- Processing speed (more tokens take longer to process)
Encoding Types
JTokkit provides several encoding types that match OpenAI's tokenizers:
| Encoding | Description | Used By |
|---|---|---|
cl100k_base |
Tokenizer of GPT-4, GPT-4 Turbo, GPT-3.5 Turbo and the text-embedding-3 / ada-002 embedding models | GPT-4, GPT-4 Turbo, GPT-3.5 Turbo, text-embedding-3-small/large, text-embedding-ada-002 |
p50k_base |
Improved vocabulary with better handling of code | GPT-3 davinci, curie, babbage, ada; Codex models |
p50k_edit |
Specialized for edit models | text-davinci-edit-001, code-davinci-edit-001 |
r50k_base |
Original GPT-3 tokenizer with basic vocabulary | Older davinci, curie, babbage, ada models |
o200k_base |
Current tokenizer of OpenAI's GPT-5, GPT-4.1, GPT-4o and o-series models (per OpenAI's tiktoken) | GPT-5, GPT-4.1, GPT-4o, GPT-4o-mini, o1, o3, o4-mini |
Models and Context Windows
Each OpenAI model has a maximum context window (number of tokens it can process in one request):
| Model | Max Tokens | Encoding Type |
|---|---|---|
gpt-5 |
272,000 | o200k_base |
gpt-4.1 |
1,047,576 | o200k_base |
gpt-4o |
128,000 | o200k_base |
gpt-4o-mini |
128,000 | o200k_base |
o3 |
200,000 | o200k_base |
o4-mini |
200,000 | o200k_base |
o1 |
200,000 | o200k_base |
gpt-4-turbo |
128,000 | cl100k_base |
gpt-4 |
8,192 | cl100k_base |
gpt-3.5-turbo |
16,385 | cl100k_base |
text-embedding-3-large |
8,191 | cl100k_base |
text-embedding-3-small |
8,191 | cl100k_base |
text-embedding-ada-002 |
8,191 | cl100k_base |
Token Counting Methods
This calculator offers two counting methods:
| Method | Standard | Ordinary |
|---|---|---|
encode() |
Full tokenization that returns token IDs with special handling for HTML | encodeOrdinary() treats HTML as plain text |
countTokens() |
Optimized method that only counts tokens with special handling for HTML | countTokensOrdinary() treats HTML as plain text |
HTML handling: When your text contains HTML, standard methods recognize and process tags specially, while ordinary methods treat HTML tags as regular text. This can result in different token counts. Use ordinary methods if you want your HTML to be preserved exactly as written.
Token Counting Tips
- Repetitive text often uses fewer tokens than unique text of the same length
- Whitespace (spaces, tabs, newlines) counts as tokens
- Special characters may be tokenized individually, increasing token count
- Common words typically use fewer tokens than rare or technical terms
- Non-English text often uses more tokens than equivalent English text
- Code can be tokenized efficiently with the newer tokenizers (o200k_base, cl100k_base)
Common Token Count Examples
| Text | Tokens | Note |
|---|---|---|
| "Hello world" | 2 | Common words often count as single tokens |
| "antidisestablishmentarianism" | 5 | Long words split into multiple tokens |
| "こんにちは" | 3 | Non-Latin characters often need more tokens |
| "1234567890" | 3 | Numbers are often chunked into multiple tokens |
| "<div>Hello</div>" | 4-6 | HTML tags may count differently between standard and ordinary methods |
Interpreting Your Token Count
A token is the unit a model actually reads and writes: a whole word, part of a word, a punctuation mark, or a run of whitespace. The count shown is the length of the list produced when your text is encoded with the selected encoding, as in the formula \(\text{Tokens} = \operatorname{len}(\operatorname{encode}(\text{Text}))\).
Context window
Every model has a context window, which is the maximum number of tokens it can handle in one request. Input and output share that limit. The prompt you send and the reply the model generates both count toward it. If your prompt uses most of the window, little room is left for the response. Some APIs also let you set a separate cap on output length.
Billing
OpenAI's API bills per token, usually with separate rates for input and output tokens. Cost therefore scales roughly linearly with token count: doubling the tokens roughly doubles that part of the cost. Check OpenAI's current pricing page for actual rates.
Counts differ between encodings
Each model family uses its own encoding (tokenizer vocabulary). For example, GPT-4 and GPT-3.5-turbo use cl100k_base, while GPT-5, GPT-4.1, GPT-4o, GPT-4o-mini and the o-series (o1, o3, o4-mini) use o200k_base. Older models such as the davinci family use p50k_base or r50k_base. Because the vocabularies differ, the same text can produce different totals. Text in many non-English languages and some code often tokenizes into fewer tokens under o200k_base, while plain English often differs little. Always count with the encoding that matches the model you will call.
Chat-format overhead
This tool counts the raw text you paste. When you use a chat API, each message is wrapped with role and delimiter tokens, adding a few tokens per message, and the reply is also primed with a few extra tokens. Those are not included in a raw-text count, so the real prompt total for a multi-message conversation is slightly higher than the sum of the message texts. The exact overhead depends on the model and message format.
Token Counts Across Encodings
The samples below are illustrative. Counts were derived from the standard tiktoken vocabularies for these short inputs. Your own text, especially longer text, may behave differently, so use the tool above for exact counts.
| Sample input | Characters | cl100k_base tokens | o200k_base tokens | Characters per token (cl100k_base) |
|---|---|---|---|---|
| Hello, world! | 13 | 4 | 4 | 3.25 |
| The quick brown fox jumps over the lazy dog. | 44 | 10 | 10 | 4.4 |
| 1234567890 | 10 | 4 (123, 456, 789, 0) | 4 (123, 456, 789, 0) | 2.5 |
Short, common English text usually comes out near 4 characters per token, and it is the same under both encodings here. Digits are split into groups of up to three, so long numbers use many tokens per character.
Differences between encodings tend to show up more with non-English scripts, source code, and emoji. These can take several tokens per character, and o200k_base often needs fewer tokens for them than cl100k_base. Paste such text into the tool and switch the encoding to compare the totals directly.