“Sometimes the numbers are correct, but the conclusion is wrong.”

This article is inspired by the philosophy of explanation in Michael Nielsen’s essay, Reinventing Explanation, on building intuition rather than merely presenting facts. So instead of jumping straight to the definition, let’s start by feeling why Simpson’s Paradox is surprising — and only then explain it.

Have you ever seen this happen?

Imagine two coaching institutes preparing students for the GATE examination.

The newspaper reports the overall pass percentage:

Most people would immediately say, “Institute A is clearly better.”

Now suppose someone shows another table — this time splitting students by category:

Now things get confusing. Institute B performs better in both categories. Yet when all students are combined, Institute A still has the higher overall pass percentage.

How can that be? Did someone make a mistake? Did the newspaper calculate the percentages wrongly? Can both tables be correct at the same time?

Take a minute before reading further.

Most people — including many professors the first time they see it — feel that this should be impossible. But it isn’t. This is called Simpson’s Paradox.

The surprise

Simpson’s Paradox is one of those ideas that changes how you look at data forever. It teaches a simple lesson:

The overall average does not always tell the real story. Sometimes the data inside different groups tells one story. When those groups are merged together, the story changes completely.

The mathematics is perfectly correct. Our intuition is what fails.

Why does this happen?

Imagine there are only two kinds of students: those who are already very strong, and those who need more help.

Now suppose Institute A admits mostly strong students, while Institute B admits many more weaker students. Even if Institute B teaches both groups slightly better, its overall result can still come out lower — because a much larger fraction of its students started out at a disadvantage.

Think of cricket. Suppose two batsmen play on different pitches — one mostly bats on flat, easy batting tracks, the other usually bats on difficult green pitches. Simply comparing their overall batting averages may not tell you who actually bats better under similar conditions. The mix of situations matters. That hidden mix is what creates Simpson’s Paradox.

A simple visual way to think about it

Imagine two baskets of mangoes. Basket A contains mostly ripe mangoes. Basket B contains mostly unripe mangoes.

Now compare two farmers. Farmer B actually grows slightly better mangoes than Farmer A — in both baskets. But Farmer B happened to send many more unripe mangoes to the market. Overall, Farmer A’s mangoes may appear better.

The problem isn’t the quality. The problem is that we mixed together two very different groups.

Example from an Indian college

Suppose two teachers teach Programming. Teacher A teaches one section, Teacher B teaches another. The final university results are:

So everyone says, “Teacher A is better.”

Now the principal separates students based on their Class 12 mathematics marks:

Students with strong mathematics

Students with weaker mathematics

Teacher B performs better in both groups. So why is Teacher A’s overall result higher? Because Teacher A happened to receive many more students who already had strong mathematics backgrounds, while Teacher B had a much larger number of weaker students. The overall percentage is therefore influenced by who entered the class — not only by how well they were taught.

Another everyday example

Suppose two hospitals perform surgeries. Hospital A mostly receives patients with minor health problems. Hospital B receives many complicated emergency cases.

Even if Hospital B performs better for both easy and difficult surgeries separately, its overall success rate could still look lower — simply because it treats far more difficult cases. Without knowing the type of patients each hospital receives, comparing only the overall success rate would be misleading.

What is really happening?

A hidden variable changes everything. In our examples, the hidden variable was: academic background, patient condition, exam difficulty, starting ability.

When we ignore these factors and simply average everything together, we may reach the wrong conclusion. Statisticians call these hidden influences confounding variables. You don’t need to remember the term — just remember the idea:

Always ask whether different kinds of data have been mixed together.

Does this mean averages are bad?

Not at all. Averages are useful. But averages answer only one question — they do not explain why the average looks the way it does.

Whenever you see an average, ask yourself:

• Who is included?
• Are all people similar?
• Are different groups being mixed?
• Would the conclusion change if I looked at each group separately?

These four questions can save you from many wrong conclusions.

Where does Simpson’s Paradox appear?

It appears surprisingly often. Researchers encounter it in medicine, education, economics, sports, machine learning, public policy, and business analytics. Whenever data from different groups is combined, Simpson’s Paradox becomes possible.

What should you learn from this?

After reading this article, here are the ideas to remember.

1. Bigger numbers do not always tell the complete story. An overall percentage can hide important details.

2. Always look inside the groups. If possible, examine the data separately before combining everything.

3. Ask what has been mixed together. Different student backgrounds. Different hospitals. Different departments. Different states. Different years. Mixing different populations can completely change the conclusion.

4. Statistics is not only about calculation. It is also about asking good questions. Sometimes the most important question is not “What is the average?” — instead it is “Average of whom?”

A small thought experiment

Suppose your college announces: “The placement percentage has increased this year.” Before celebrating, ask:

• Did all departments improve?
• Or did one department become much larger?
• Did weaker departments shrink?
• Were more students eligible this year?

The overall number may be true. But the complete story may still be hidden. That is exactly the lesson of Simpson’s Paradox.

Conclusion

Simpson’s Paradox reminds us that data can be honest while our interpretation is mistaken. It teaches an important habit that every engineer, scientist, economist, and manager should develop: never stop at the overall average. Look inside the data. Understand the groups. Ask what has been combined. Only then can numbers reveal the truth instead of hiding it.

As the famous statistician John Tukey once said, “The greatest value of a picture is when it forces us to notice what we never expected to see.” Simpson’s Paradox does exactly that — it forces us to question our first impression and think more carefully about what the data is really saying.

Part 1 built up the vocabulary for a single random variable: the information content \(\log_2 \frac{1}{p(x)}\) of an outcome, entropy \(H(X)\) as its probability-weighted average, redundancy, cross-entropy, and KL divergence. All of it concerned one variable at a time. Most interesting questions, though, involve relationships between variables: how much does today’s weather tell us about tomorrow’s? How much do two random variables share? This post extends entropy to two (or more) variables, following the same two sources as Part 1 — Olah’s essay and Pinkard & Waller’s tutorial — and ends with the entropy rate, which generalizes entropy to sequences that are not independent and identically distributed.

Joint entropy

The simplest extension is to ask how much information is needed to communicate the outcome of two variables together. If we flatten the joint distribution \(p(x,y)\) over the product space \(\mathcal{X} \times \mathcal{Y}\) and treat it as a distribution over a single combined variable, entropy applies unchanged:

This is the joint entropy — the average number of bits needed to communicate both \(X\) and \(Y\) using a code built for their joint distribution . When \(X\) and \(Y\) are independent, \(p(x,y) = p(x)p(y)\), and substituting this in shows \(H(X,Y) = H(X) + H(Y)\): independent information simply adds. When they are dependent, the joint entropy is less than the sum of the individual entropies — some information is shared, and we shouldn’t have to pay for it twice. Quantifying exactly how much is shared is the job of mutual information, below.

Conditional entropy

Suppose you already know \(Y\). How much more information do you need, on average, to also learn \(X\)? This is the conditional entropy \(H(X \mid Y)\): the uncertainty remaining in \(X\) once \(Y\) is known.

Start at a single point: if we already know \(Y = y\), the information needed to learn \(X\) is the entropy of the conditional distribution \(p(x \mid y)\),

Averaging over all possible values of \(y\), weighted by how likely each is, gives the conditional entropy in its usual form:

A concrete example makes this tangible. Pinkard and Waller consider drawing objects with both a color (\(X\)) and a shape (\(Y\)) at random . Suppose that, conditioned on the shape being a particular one (say \(\bullet\)), the color is equally likely to be green, yellow, or blue. Then \(H(X \mid Y = \bullet) = \log_2 3 \approx 1.58\) bits: three equally probable outcomes remain even after learning the shape. If instead conditioning on a different shape narrows the color down to a single possibility, the conditional entropy for that value of \(Y\) is exactly zero — no uncertainty is left.

Conditional entropy can never exceed the unconditional entropy: \(H(X \mid Y) \le H(X)\), with equality iff \(X\) and \(Y\) are independent. Combined with joint entropy, this gives a clean chain of inequalities,

and an identity that will anchor the next section: \(H(X,Y) = H(Y) + H(X \mid Y)\) — the information in both variables is the information in one, plus whatever’s left in the other after conditioning on it.

Mutual information

Joint entropy measures the total information in \(X\) and \(Y\) together; conditional entropy measures what’s left in one after learning the other. The information they have in common — how much observing \(Y\) reduces our uncertainty about \(X\) — is the mutual information:

The intuition is a counting argument: \(H(X) + H(Y)\) counts the shared information twice (once as part of \(X\), once as part of \(Y\)), while \(H(X,Y)\) counts it once. The difference is exactly one copy of what’s shared .

Pointwise mutual information

Just as entropy has a pointwise predecessor (information content), mutual information has a pointwise version. For a specific pair of outcomes \((x,y)\), the pointwise mutual information is

the surprise of \(x\) on its own, minus the (typically smaller) surprise of \(x\) once \(y\) is already known . Averaging this quantity over the joint distribution recovers the usual definition,

This expression is worth pausing on, because it is exactly the KL divergence from Part 1 — specifically, the KL divergence between the true joint distribution \(p(x,y)\) and the “naive” product-of-marginals distribution \(p(x)p(y)\) that we’d get by (incorrectly) assuming independence . Mutual information is the number of bits you save by understanding the real relationship between \(X\) and \(Y\), instead of assuming they’re independent. Two useful properties follow immediately: mutual information is symmetric, \(I(X;Y) = I(Y;X)\), and it is always non-negative (since KL divergence is), with \(I(X;Y) = 0\) exactly when \(X\) and \(Y\) are independent.

How the pieces fit together

Joint entropy, conditional entropy, and mutual information are not independent concepts — they are different ways of partitioning the same total information, and it helps to see all four quantities laid out together, with area representing bits.

Reading the diagram: \(H(X,Y) = H(X) + H(Y\mid X) = H(Y) + H(X \mid Y)\), and the mutual information is exactly the amount by which \(H(X)\) and \(H(Y)\) overlap. All three of the following are algebraically equivalent expressions for mutual information, and each has its own reading:

The first says mutual information is how much \(Y\) shrinks our uncertainty about \(X\); the second is the same statement with the roles reversed (consistent with symmetry); the third is the counting argument from before.

Variation of information

If mutual information measures what two variables share, its complement measures what they don’t: the variation of information,

Variation of information is a genuine metric — symmetric, non-negative, and satisfying the triangle inequality — on the space of jointly distributed variables . It is zero exactly when knowing one variable tells you the other completely, and it grows as the variables become more independent. It’s worth contrasting this with KL divergence from Part 1: KL divergence measures distance between two distributions over the same variable(s); variation of information measures distance between two jointly distributed variables, within a single distribution.

Fractional bits, revisited

Part 1 flagged something odd: optimal codeword lengths are frequently fractional, which seems meaningless for a single message. Consider a distribution with two outcomes, \(a\) (probability 71%) and \(b\) (probability 29%). The ideal codeword lengths are \(\log_2\frac{1}{0.71}\approx 0.49\) bits for \(a\) and \(\log_2\frac{1}{0.29}\approx 1.79\) bits for \(b\) — neither a whole number .

If we must send a single symbol, we’re forced to round, giving an average length of 1 bit (using codewords 0 and 1) rather than the entropy \(H \approx 0.87\) bits. But if we encode two draws jointly, something better happens. The four two-symbol outcomes have probabilities \(p(aa) = 0.504\), \(p(ab) = p(ba) = 0.206\), and \(p(bb) = 0.084\); assigning shorter codewords to the more probable pairs and rounding gives an average of about 1.8 bits for two symbols — 0.9 bits per symbol, already better than sending them independently (2 bits for two symbols) . As the number of jointly-encoded symbols \(N \to \infty\), the rounding overhead per symbol vanishes and the achievable rate approaches the entropy exactly. This is a special case of the AEP from Part 1: block-encoding the typical set of a large-\(N\) sequence gets arbitrarily close to the entropy bound.

(In practice, Huffman coding — essentially the scheme sketched here — needs this kind of symbol-grouping to approach the entropy limit gracefully; arithmetic coding handles fractional bits natively and is asymptotically optimal without grouping .)

Entropy rate of a stochastic process

Everything so far has assumed a sequence of independent and identically distributed random variables. Real sequences — text, weather, stock prices — are rarely independent: today’s value constrains tomorrow’s. An ordered sequence of random variables \(\mathbf{X} = X_1, X_2, \ldots\), without an independence assumption, is called a stochastic process .

Two equivalent definitions

In the i.i.d. case, entropy is additive: \(H(X_1, \ldots, X_N) = N \cdot H(X)\), so the entropy rate — entropy per symbol — is just \(H(X)\). For a general stochastic process, we need a definition that doesn’t assume this factorization. There are two natural candidates.

The first treats entropy rate as the average uncertainty per draw, over all \(N\) draws jointly:

The second treats it as the uncertainty about the next draw, given everything before it:

For a stationary process — one whose joint distribution is shift-invariant, \(p_{X_1,\ldots,X_N} = p_{X_{1+k},\ldots,X_{N+k}}\) for any \(k\) — both definitions converge to the same value as \(N \to \infty\) .

Consider a “magical urn” version of our marble example: the first draw is uniform over four colors (\(H(X_1) = 2\) bits), but every draw after that is more likely to repeat the previous color — \(p(\text{same color}) = \tfrac58\), and \(\tfrac18\) for each of the other three colors, independent of \(N\) (a first-order Markov chain). Because each color is still marginally uniform, \(H(X_k) = 2\) bits for every \(k\) in isolation — but the joint entropy of consecutive draws is less than the sum of their individual entropies, because knowing \(X_N\) substantially reduces uncertainty about \(X_{N+1}\) . Concretely,

constant for every \(N \ge 1\), well below the unconditional 2 bits of the first draw. The two definitions of entropy rate — averaging the joint entropy, and the conditional entropy of the next draw given the past — both converge to this same \(\approx 1.549\) bits as \(N\) grows, exactly as stationarity predicts:

Redundancy of a stochastic process

Moving from an i.i.d. sequence to a general stochastic process also forces us to revisit redundancy from Part 1, since it can no longer be defined variable-by-variable. The general definition accounts for the joint entropy against the maximum entropy of the full product space :

In the marble-urn example, every individual draw is marginally uniform, so \(H(X_k) = H_{\max}(\mathcal{X})\) for each \(k\) in isolation — there is no redundancy variable-by-variable. But the joint distribution is far from the maximum-entropy (independent, uniform) joint distribution, because the outcome of one draw is highly informative about its neighbor. This is exactly why we can guess the next color far better than chance, given the previous one: the redundancy lives in the higher-order dependencies between draws, not in any single draw’s marginal distribution.

Conclusion

We’ve now built, from first principles, the core quantities of information theory: information content and entropy (Part 1); cross-entropy and KL divergence (Part 1); joint entropy, conditional entropy, and mutual information (this post); and the entropy rate of a stochastic process (this post). Two threads run through all of it. First, every one of these quantities has a coding interpretation — the expected length of some code under some scenario — which is what makes entropy feel inevitable rather than an arbitrary formula. Second, KL divergence quietly underlies more than it first appears to: cross-entropy is entropy plus a KL term, and mutual information is a KL divergence, between a joint distribution and the product of its marginals.

This is deliberately not the whole of information theory. Both source texts continue well beyond what we’ve covered here: Pinkard and Waller extend these ideas to continuous random variables and differential entropy, and go on to channel capacity and lossy compression — the second of the “two key problems” their tutorial sets out to address . Olah’s essay closes with pointers to error-correcting codes and the surprising appearances of entropy in quantum information, thermodynamics, and gambling . For those directions, Shannon’s original paper and Cover & Thomas’s Elements of Information Theory — the standard reference recommended by both of our sources — are the natural next stop.

In the 1940s, Claude Shannon gave the informal idea of “information” a precise mathematical footing. The resulting theory answers questions that feel intuitive but are hard to pin down without it: How uncertain am I about an outcome? How much does knowing one thing tell me about another? How different are two beliefs about the world?

This two-part post builds up the core machinery of information theory — entropy, cross-entropy, KL divergence, joint and conditional entropy, mutual information, and the entropy rate of a stochastic process — visually and rigorously at the same time. It closely follows two sources. The first is Christopher Olah’s essay Visual Information Theory, which builds intuition for entropy and related quantities through codes and areas rather than through the sums directly . The second is Pinkard and Waller’s A visual introduction to information theory, a more recent, notation-careful tutorial that formalizes many of the same ideas with figures grounded in a simple marble-drawing experiment . Part 1 (this post) covers single-variable quantities: information content, entropy, redundancy, typical sequences, cross-entropy, and KL divergence. Part 2 covers quantities involving two or more variables — joint entropy, conditional entropy, mutual information — and closes with the entropy rate of a stochastic process.

Throughout, we assume only familiarity with basic probability: random variables, joint and conditional distributions, and expectation.

Information: reducing uncertainty

Gaining information means having your uncertainty about something reduced. If I tell you the outcome of a coin flip you haven’t seen, I’ve resolved all your uncertainty about it. If I tell you something you already knew for certain, I’ve given you no information at all.

Consider drawing a marble at random from an urn containing marbles of four colors, with replacement, where

Suppose you learn that the marble drawn is not blue. That single fact rules out half of the probability mass, leaving a distribution renormalized over the remaining three colors. Learning a fact that was less likely to be true rules out more probability mass, and hence carries more information — this is precisely why rare events are more informative than common ones .

We can make this quantitative. If ruling out half the probability mass corresponds to one unit of information, then the information content of an outcome \(x\) with probability \(p(x)\) is

measured in bits. An outcome with probability \(\tfrac12\) carries exactly 1 bit; an outcome with probability \(\tfrac18\) carries 3 bits, because learning it happened is equivalent to eliminating three successive halvings of the probability mass. The entropy of a random variable \(X\) is the probability-weighted average information content of its outcomes:

Entropy is the expected number of bits of surprise you get, on average, from observing \(X\). It is maximized when probability is spread as evenly as possible across outcomes, and it is zero exactly when the outcome is certain in advance (\(p(x) = 1\) for some \(x\)) .

Visualizing joint distributions

Before going further, it helps to have a way of picturing distributions over two variables at once . Suppose we track the weather (sunny 75% of the time, raining 25%) and clothing (t-shirt 62% of the time, coat 38%). If the two are independent — knowing the weather tells us nothing about clothing choice — the joint distribution factors as \(p(x,y) = p(x)\,p(y)\), and a grid of the joint probabilities looks like a simple product of the two marginals: every row is a scaled copy of every other row.

When the variables interact, some cells of the grid swell with extra probability (it’s more likely to wear a coat when it’s raining) at the expense of others. The fundamental identity connecting joint and conditional probability, \(p(x,y) = p(x)\cdot p(y\mid x)\), lets us factor any joint distribution one variable at a time — and it is the seed from which joint entropy and conditional entropy (Part 2) will grow.

Entropy: the cost of encoding

Entropy has a second, equally important interpretation: it is the shortest possible average length, in bits, of a code for a sequence of outcomes from a distribution . This is worth deriving carefully, because the derivation is what makes the formula feel inevitable rather than arbitrary.

Suppose we want to communicate a sequence of words drawn from a small vocabulary — say, an imaginary friend who only ever says “dog,” “cat,” “fish,” or “bird,” with probabilities \(\tfrac12, \tfrac14, \tfrac18, \tfrac18\) . A code assigns each word a binary codeword; to send a message we concatenate the codewords for each word in sequence.

Prefix codes and the space of codewords

If every codeword has the same length, decoding is trivial — split the bitstream every \(k\) bits. But we’d like common words (like “dog”) to get short codewords, so the average message is short. This creates a subtlety: with variable-length codewords, how does the receiver know where one codeword ends and the next begins?

The answer is the prefix property: no codeword may be a prefix of another. A code with this property is called a prefix code, and it is always uniquely decodable — you can read a bitstream left to right and unambiguously identify each codeword as soon as its pattern is complete.

The prefix property has a cost. Choosing the codeword 01 forbids every longer codeword that starts with 01 — 010, 0110101, and so on — because they would be ambiguous with it. A quarter of all possible bitstrings begin with 01, so choosing it as a codeword “spends” a quarter of the total space of possible codewords. In general, a codeword of length \(L\) costs \(2^{-L}\) of the total space .

The following binary tree shows one valid prefix code for the four-word vocabulary: 0 for dog, 10 for cat, 110 for fish, 111 for bird. Every codeword is a leaf, and no codeword is an ancestor of another — that’s the prefix property, drawn as a tree.

The optimal codeword length

Think of building a code as spending a fixed budget: buying a codeword of length \(L\) costs \(2^{-L}\) of the total space of codewords, and using it costs us \(p(x) \cdot L\) extra bits in our average message length, since it’s used a \(p(x)\) fraction of the time. The natural strategy — spend a fraction \(p(x)\) of the budget on the codeword for \(x\) — turns out to be optimal, not merely reasonable . Olah proves this with a marginal argument: at the natural allocation, the benefit-to-cost ratio of shortening any single codeword is exactly 1, the same for every codeword; perturbing away from it (spending \(\epsilon\) more on one codeword and \(\epsilon\) less on another) unbalances the ratios and creates an incentive to shift back. Since this holds for every pair of codewords, the natural allocation cannot be improved.

If we spend \(p(x)\) of the budget on the codeword for \(x\), and a codeword of length \(L\) costs \(2^{-L}\), then solving \(2^{-L} = p(x)\) for \(L\) gives the optimal codeword length:

This is exactly the information content of \(x\) from the previous section — no coincidence. The optimal code assigns each outcome a codeword whose length equals its information content in bits.

Putting it together

The average codeword length under the optimal code is, by definition, the entropy:

For our dog/cat/fish/bird example this works out to \(\tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac18(3) = 1.75\) bits — and no code, however clever, can do better on average . This is a genuine lower bound: entropy is the shortest possible average encoding length for a sequence of outcomes from a given distribution, a fact known as the source coding theorem.

One wrinkle: optimal codeword lengths are frequently fractional (e.g. \(\log_2 \tfrac{1}{0.71} \approx 0.49\) bits), which is meaningless for a single codeword — you can’t send half a bit. But if you encode several draws jointly, ideal lengths add, and the rounding overhead per event shrinks toward zero as the number of jointly-encoded events grows . There is a real sense in which fractional bits are achievable on average, even though no single message can have a fractional length; we will use this idea again in Part 2.

Redundancy and maximum entropy

Entropy is maximized when probability is spread as evenly as possible over the outcome space \(\mathcal{X}\). In that case,

Redundancy quantifies how much shorter our messages become because the distribution is not uniform — the more concentrated the probability mass, the larger the redundancy, and the more compressible the source. A deterministic variable (\(p(x)=1\) for one outcome) has zero entropy and maximum possible redundancy; a uniform variable has zero redundancy, because there is nothing left to exploit.

Typical sequences and the AEP

Entropy governs a length-\(N\) i.i.d. sequence too: the total information content of \(N\) independent draws is \(N \cdot H(X)\) on average. But individual sequences vary enormously in actual information content — an all-blue sequence (the single most probable outcome) can be far cheaper to encode than a “typical” one, and far more expensive than a maximally rare one .

A typical sequence is one whose information content is close to this average: for small \(\epsilon > 0\),

As \(N \to \infty\), the asymptotic equipartition property (AEP) kicks in: almost all of the probability mass concentrates onto the typical set, regardless of how small \(\epsilon\) is, and every typical sequence has probability \(\approx 2^{-NH(X)}\). Since probabilities sum to 1, there must be \(\approx 2^{NH(X)}\) typical sequences. This is precisely what makes lossless compression at rate \(H(X)\) achievable: a scheme that assigns a unique length-\(NH(X)\) binary string to every typical sequence, and ignores the (vanishingly probable) rest, is lossless in the limit — and no scheme can reliably do better .

Cross-entropy

Now suppose two people communicate using the same words but different frequencies. Bob mostly talks about dogs; his wife Alice mostly talks about cats. Bob’s code — optimized for his own distribution \(p\) — is suboptimal when Alice uses it to encode her distribution \(q\), because it assigns short codewords to words Alice rarely uses .

The average message length when encoding events from \(q\) using the code optimized for \(p\) is the cross-entropy:

Cross-entropy is not symmetric — \(H_p(q) \neq H_q(p)\) in general — and this asymmetry is not a technicality; it is the whole point. \(H_q(p)\) is large exactly when there’s an outcome common under \(p\) but rare under \(q\): that outcome gets an unnecessarily long codeword, which hurts badly because \(p\) uses it often. Whether the “large” direction is \(H_p(q)\) or \(H_q(p)\) depends on which distribution is doing the frequent using and which one built the code.

Cross-entropy is always at least the entropy of the distribution being encoded, \(H_p(q) \ge H(q)\), with equality iff \(p = q\). This makes it a natural, if asymmetric, measure of how different two distributions are — and it is why cross-entropy is the workhorse loss function for classification in machine learning: minimizing \(H_p(q)\) where \(q\) is the true label distribution and \(p\) is the model’s predicted distribution directly penalizes the model for being confidently wrong .

KL divergence

The excess length caused by using the wrong code — the gap between cross-entropy and entropy — is the Kullback–Leibler (KL) divergence:

KL divergence is zero exactly when \(p = q\) and grows as the two distributions diverge, which is why it behaves like a “distance” between distributions (though it is not symmetric and does not satisfy the triangle inequality, so it is not a metric in the strict sense) . It shows up constantly whenever we want one distribution to be close to another — variational inference, information geometry, and, again, machine learning, where minimizing \(D_q(p)\) between a predicted and target distribution is equivalent to minimizing cross-entropy alone (since \(H(p)\) doesn’t depend on the model).

What’s next

We now have the vocabulary for a single random variable: information content, entropy, redundancy, cross-entropy, and KL divergence. Part 2 extends all of this to two or more variables — joint entropy, conditional entropy, and mutual information — and finishes with the entropy rate of a stochastic process, the natural generalization of entropy to sequences that aren’t independent and identically distributed.

For a fuller treatment than either source attempts, Shannon’s original paper remains remarkably readable , and Cover & Thomas’s Elements of Information Theory is the standard graduate reference .

If you prefer publishing blog posts on medium.com or other external sources, starting version v0.5.0, al-folio lets you to display your external posts in the blog feed of your website! 🎉🎉

Configuring external sources of super simple. After upgrading to v0.5.0, just add the following section to your _config.yml:

external_sources:
  - name: medium.com  # name of the source (arbitrary string)
    rss_url: https://medium.com/@<your-medium-username>/feed

The example above adds your medium.com blog post feed as an external source. But you can add arbitrary RSS feeds as sources.

Any questions or suggestions? 👉 Start a discussion on GitHub!