What is Shannon Entropy?
Shannon entropy measures the average amount of uncertainty, or information, contained in a probability distribution. Introduced by Claude Shannon in 1948, it is the foundation of information theory and is widely used in data compression, cryptography, machine learning, and statistics. A distribution where one outcome is certain has zero entropy, while a uniform distribution over many outcomes has the maximum possible entropy.
How to Use This Calculator
Enter the probabilities of each outcome as a comma-separated list, for example 0.5, 0.25, 0.25. Choose your logarithm base: base 2 gives the answer in bits, base e gives nats, and base 10 gives dits (hartleys). If your probabilities do not sum exactly to 1, the calculator automatically normalizes them by dividing by their total. Zero or negative entries are ignored.
The Formula Explained
The entropy is calculated as $$H = -\sum_{i} p_i \, \log_{b} p_i.$$ Each probability contributes \(-p_i \cdot \log_{b}(p_i)\); summing these terms and negating gives a non-negative value. The maximum entropy for \(n\) outcomes is \(\log_{b}(n)\), achieved when all outcomes are equally likely. Efficiency expresses your entropy as a percentage of that maximum.
Worked Example
Consider probabilities 0.5, 0.25, 0.25 in bits. The terms are \(-0.5 \cdot \log_{2}(0.5) = 0.5\), \(-0.25 \cdot \log_{2}(0.25) = 0.5\), and again 0.5. Adding them gives $$H = 1.5 \text{ bits}.$$ The maximum for 3 outcomes is \(\log_{2}(3) \approx 1.585\) bits, so efficiency \(\approx 94.64\%\).
FAQ
Why is entropy zero for a sure event? If one outcome has probability 1, there is no uncertainty, so no information is gained when it occurs.
What is the difference between bits and nats? They are just different units, set by the log base. \(1 \text{ nat} \approx 1.4427 \text{ bits}\).
Do probabilities have to sum to 1? Ideally yes, but this tool normalizes them automatically if they do not.
