What Is Shannon Entropy?
Shannon entropy measures the average amount of uncertainty, surprise, or information contained in a random variable. Introduced by Claude Shannon in 1948, it is the foundation of information theory and is measured in bits when using a base-2 logarithm. An entropy of 1 bit corresponds to the uncertainty of a single fair coin flip.
How to Use the Calculator
Enter the probability of each possible outcome separated by commas or spaces (for example 0.5, 0.25, 0.25). You can also enter raw frequencies or counts (such as 10, 5, 5) — the calculator normalizes them into probabilities automatically by dividing each value by the total. Zero and negative values are ignored. The tool returns the entropy in bits, the maximum possible entropy, and the distribution's efficiency.
The Formula Explained
The entropy is computed as $$H = -\sum_{i=1}^{n} p_i \log_2 p_i$$ summed over every outcome \(i\). Each term weights the information content of an outcome, \(-\log_2 p_i\), by how often it occurs, \(p_i\). Rare events carry more information; certain events (\(p_i = 1\)) carry none. The maximum entropy for \(n\) outcomes is \(\log_2 n\), achieved when all outcomes are equally likely. Efficiency expresses \(H\) as a percentage of that maximum.
Worked Example
Consider the distribution {0.5, 0.25, 0.25}. The entropy is: $$-[0.5\cdot\log_2(0.5) + 0.25\cdot\log_2(0.25) + 0.25\cdot\log_2(0.25)] = -[0.5\cdot(-1) + 0.25\cdot(-2) + 0.25\cdot(-2)] = 0.5 + 0.5 + 0.5 = 1.5 \text{ bits}$$ The maximum entropy for 3 outcomes is \(\log_2(3) \approx 1.585\) bits, giving an efficiency of about 94.64%.
FAQ
Why bits? Using log base 2 gives entropy in bits, the natural unit for digital information. Base e gives "nats" and base 10 gives "hartleys".
Do my probabilities have to sum to 1? No — the calculator normalizes any positive values, so you can paste raw counts directly.
What is the maximum entropy? For \(n\) equally likely outcomes it equals \(\log_2 n\). A fair coin (\(n=2\)) has max entropy of 1 bit; a fair die (\(n=6\)) about 2.585 bits.
