What Is the TF-IDF Calculator?
TF-IDF (Term Frequency–Inverse Document Frequency) is a numerical statistic used in information retrieval, search engines, and text mining to measure how important a word is to a single document within a larger collection of documents (called a corpus). This calculator lets you enter a document, a target term, the total number of documents in your corpus, and how many of those documents contain the term. It then computes the term frequency (TF), the inverse document frequency (IDF), and the combined TF-IDF score.
How to Use It
- Document: Paste the text of the document you want to analyze.
- Term: Enter the single word or phrase you want to score.
- Corpus size: The total number of documents in your collection (\(N\)).
- Documents with the term: How many documents contain the term at least once.
The calculator counts how often the term appears in your document and divides it by the total word count, then combines that with the rarity of the term across the corpus.
The Formula Explained
The standard TF-IDF formula is:
$$\begin{gathered} \text{TF-IDF} = \text{TF} \times \text{IDF} \\[1.5em] \text{where}\quad \left\{ \begin{aligned} \text{TF} &= \dfrac{\text{count of }\text{Term}\text{ in }\text{Document}}{\text{total words in }\text{Document}} \\[0.6em] \text{IDF} &= \ln\!\left(1 + \dfrac{\text{Corpus Size}}{\text{Docs with Term} + 1}\right) \end{aligned} \right. \end{gathered}$$- TF = (number of times the term appears in the document) ÷ (total words in the document)
- IDF = \(\log(N \div \text{number of documents containing the term})\)
- TF-IDF = \(\text{TF} \times \text{IDF}\)
A high TF-IDF score means the term appears frequently in your document but rarely across the corpus — a strong signal that the word is distinctive and meaningful for that document.
Worked Example
Suppose your document has 100 words and the term "marketing" appears 5 times. Your corpus contains 1,000 documents, and 100 of them mention "marketing".
- TF = \(5 \div 100 = 0.05\)
- IDF = \(\log(1000 \div 100) = \log(10) = 1\) (base 10)
- TF-IDF = \(0.05 \times 1 = 0.05\)
If "marketing" appeared in only 10 documents instead, IDF would rise to \(\log(100) = 2\), doubling the score to 0.10 — rewarding rarer, more distinctive terms.
Key Terms and Variables
| Term | Symbol | Definition |
|---|---|---|
| Term | \(t\) | The word, stem or phrase being scored. Whatever counts as one term must be defined consistently across the whole corpus. |
| Document | \(d\) | One unit of text being scored — an article, a page, a product description, a support ticket. TF is always relative to a single document. |
| Corpus | \(D\) | The full collection of documents against which rarity is judged. IDF is a property of the corpus, not of the document. |
| Corpus size | \(N\) | The number of documents in the corpus, including the one being scored. |
| Document frequency | \(df\) | How many documents contain the term at least once. A count of documents, never of occurrences, so \(0 \le df \le N\). |
| Raw term count | \(tf\) | The number of times the term occurs inside the one document — the unnormalised count. |
| Term frequency | TF | The normalised count, \(tf\) divided by the document's total word count. A proportion between 0 and 1 that removes the effect of document length. |
| Inverse document frequency | IDF | A logarithmic measure of global rarity. Large when the term appears in few documents, small when it appears in many. |
| TF-IDF | — | The product \(\text{TF} \times \text{IDF}\): high when a term is locally frequent and globally rare. |
| Tokenisation | — | Splitting raw text into countable tokens — typically lowercasing, removing punctuation, and deciding how to treat numbers, hyphens and contractions. It fixes both the TF denominator and what counts as a match. |
| Stopword | — | A very common function word (the, of, and, is) that carries little topical information. Often removed before scoring, though TF-IDF suppresses such words on its own because their IDF is near the minimum. |
| Smoothing | — | The \(+1\) terms in \(\ln(1 + N/(df+1))\). The \(+1\) in the denominator prevents division by zero when \(df = 0\) (a term unseen in the corpus); the \(+1\) inside the logarithm keeps the argument above 1, so the result is always strictly positive and never zero or negative. |
| Log base | \(\ln\), \(\log_{10}\) | The logarithm base used in IDF. Changing base only multiplies every IDF by a constant — \(\ln x = \log_{10}x \times \ln 10 \approx 2.3026\,\log_{10}x\) — so the ranking of terms is unchanged while the absolute numbers are not. Always state which base you used. |
| Vocabulary | \(V\) | The set of distinct terms across the corpus after tokenisation. Its size is the dimensionality of the TF-IDF vector space. |
Note that TF-IDF is a family of formulas rather than a single one: variants include raw-count TF, log-normalised TF \(1 + \ln(tf)\), augmented TF, and unsmoothed or probabilistic IDF. Scores from different variants should never be compared directly.
Frequently Asked Questions
Why use a logarithm in IDF? The log dampens the effect of extremely rare terms so a single occurrence doesn't dominate the score, keeping results balanced.
What log base should I use? Base 10 and natural log (ln) are both common. The base only scales results uniformly, so comparisons within the same corpus remain valid.
What does a score of zero mean? If a term appears in every document (\(N\) = documents with term), IDF becomes \(\log(1) = 0\), making the TF-IDF score zero — the word carries no distinguishing value.
Related calculators
- Cosine Similarity Calculator — similarity between two vectors.