<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[PHUONGLIB: Data Science]]></title><description><![CDATA[Các bài viết về chủ đề Data Science]]></description><link>https://www.phuonglib.com/s/data-science</link><image><url>https://substackcdn.com/image/fetch/$s_!tUFo!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68ab7f3d-2ea9-431f-a018-11cffe5264ac_1000x1000.png</url><title>PHUONGLIB: Data Science</title><link>https://www.phuonglib.com/s/data-science</link></image><generator>Substack</generator><lastBuildDate>Thu, 19 Nov 2026 21:01:59 GMT</lastBuildDate><atom:link href="https://www.phuonglib.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Phuong Nguyen]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[phuonglib@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[phuonglib@substack.com]]></itunes:email><itunes:name><![CDATA[PHUONGLIB]]></itunes:name></itunes:owner><itunes:author><![CDATA[PHUONGLIB]]></itunes:author><googleplay:owner><![CDATA[phuonglib@substack.com]]></googleplay:owner><googleplay:email><![CDATA[phuonglib@substack.com]]></googleplay:email><googleplay:author><![CDATA[PHUONGLIB]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Granger Causality in Time Series]]></title><description><![CDATA[Granger causality in plain terms: stationarity, VAR models, six testing steps, business uses such as media mix modeling, and key pitfalls.]]></description><link>https://www.phuonglib.com/p/granger-causality-in-time-series</link><guid isPermaLink="false">https://www.phuonglib.com/p/granger-causality-in-time-series</guid><dc:creator><![CDATA[PHUONGLIB]]></dc:creator><pubDate>Tue, 17 Nov 2026 16:23:25 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!c0Ry!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!c0Ry!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!c0Ry!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!c0Ry!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!c0Ry!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two paper kites over a bright field, the orange kite slightly ahead of the teal one, with a clock face as the sun.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two paper kites over a bright field, the orange kite slightly ahead of the teal one, with a clock face as the sun." title="Two paper kites over a bright field, the orange kite slightly ahead of the teal one, with a clock face as the sun." srcset="https://substackcdn.com/image/fetch/$s_!c0Ry!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!c0Ry!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!c0Ry!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!c0Ry!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3606079f-da52-454c-9d5b-9da6e89084fb_2560x1440.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><blockquote><p><strong>Key takeaways</strong></p><ul><li><p>Two trending series will correlate even when they are unrelated; always plot, test for stationarity and difference before modeling.</p></li><li><p>Estimating effects from time series means dealing with inflation, autocorrelation, slow adjustment and, hardest of all, endogeneity.</p></li><li><p>X "Granger-causes" Y if the past of X improves forecasts of Y. It is predictive causality, not the effect of an intervention.</p></li><li><p>A six-step procedure runs from plotting the series to impulse response functions; conclusions can change with the number of lags.</p></li><li><p>Anticipation and omitted variables create spurious Granger causality, so treat results as clues for where to experiment.</p></li></ul></blockquote><div><hr></div><p>Does higher marketing spending raise profits, or do higher profits lead companies to spend more on marketing? To most managers the answer seems obvious: marketing drives sales, sales drive profit. But the reverse is just as plausible: many companies set the marketing budget as a percentage of revenue, so in good years they spend more. In a company's quarterly figures the two series move together, and a high correlation does not tell which direction is right.</p><p>Time-ordered data (weekly sales, monthly ad spend, daily prices) are the most common kind of data in business and also the most misleading. Two trends moving in the same direction over time are a classic trap that makes correlation look like causation. This article goes further: how to estimate effects from time series carefully, and what <strong>Granger causality</strong>, a widely used notion for judging which series "leads" which, can and cannot tell us. It draws mainly on the chapter on causality in time series in Haughton et al. (2026).</p><p>The article has six parts. Section 1 covers the trap of spurious correlation and the notion of stationarity. Section 2 works through an example estimating the effect of gasoline prices on sales, with the typical problems of time series. Section 3 introduces Granger causality, and Section 4 walks through six testing steps using gold and oil prices. Section 5 reviews business applications, including media mix modeling. Section 6 explains why Granger causality is not causality in the full sense.</p><h2>1. Time series and the trap of spurious correlation</h2><p>Before talking about causality, it helps to understand why time series produce spurious correlations more readily than any other kind of data. This section introduces stationarity and how to check for it.</p><h3>1.1. Ohio beavers and Irish GDP</h3><p>Haughton et al. (2026) offer an unforgettable example: over 1960&#8211;2019, the number of beaver colonies in Ohio and real GDP per capita in Ireland had a correlation of 0.88. There is no conceivable causal link between them, not even a plausible common cause. They correlate simply because both drift over time. As early as 1926, Yule (1926) asked why time series so often produce "nonsense correlations", and the answer is still the same: when two series both trend, they look related even when entirely independent.</p><h3>1.2. Stationarity and differencing</h3><p>A series is <strong>stationary</strong> when its mean and variability do not change over time: it fluctuates around a stable level rather than drifting up or down. A growing company's sales, the consumer price index and GDP are all non-stationary. The usual remedy is <strong>differencing</strong>: instead of comparing this month's sales level with this month's ad spend, compare the <em>change</em> in sales with the <em>change</em> in ad spend. When this is done for Ohio beavers and Irish GDP, the correlation disappears completely. For series that grow proportionally, differencing the logarithm gives growth rates, which are often easier to interpret.</p><p>The most common test of stationarity is the <strong>augmented Dickey-Fuller (ADF) test</strong>. If a series has a structural break (a pandemic changing shopping behavior, for example), other tests are more suitable. The practical rule is: always plot the series first, always check stationarity, and never regress one trending series on another without dealing with the trend.</p><h2>2. Estimating the effect of price from time series</h2><p>The first example in Haughton et al. (2026) is a pricing problem. A fuel company sells gasoline at $3.51 a gallon and wants to know what would happen if it raised the price. It has data since 2005 on its sales, regional GDP and population, a price index, its own gasoline price and a rival's diesel price. The example shows the typical problems of estimating effects from time series, summarized step by step below.</p><h3>2.1. Preparing the data</h3><p>Two adjustments are almost always necessary. The outcome should be sales <strong>per capita</strong>, to remove the effect of population growth. Prices and GDP should be converted to <strong>real</strong> terms (adjusted for inflation) at a base year, because $3 in 2005 and $3 in 2023 do not buy the same amount. Then plot the key series: the real prices of gasoline and diesel, real GDP per capita and gasoline sales per capita.</p><p>Next comes the causal assumption. Gasoline sales depend on income (GDP per capita), on the gasoline price (higher price, lower quantity, by the law of demand) and on the diesel price, a close substitute (when diesel is expensive, some drivers switch to gasoline cars). The crude oil price affects both gasoline and diesel prices, so it is a confounder that should be included to block the back-door path.</p><h3>2.2. Autocorrelation and ways to deal with it</h3><p>An ordinary linear regression fits reasonably well, with coefficients of the expected signs. But the Durbin-Watson statistic is well below 2, indicating <strong>autocorrelation</strong> in the residuals: a positive shock this year tends to be followed by another next year. With autocorrelation, p-values become too optimistic, meaning the model is more confident than it deserves to be. Autocorrelation often signals a missing variable or a wrong functional form.</p><p>Haughton et al. (2026) try several remedies, summarized in Table 1. Adding a time trend improves the fit but may "absorb" part of the effect that really belongs to income. A log-log model gives coefficients that are <strong>elasticities</strong>: the price elasticity of demand for gasoline is &#8722;0.097, so a 10% price rise reduces sales by about 1%. The Prais-Winsten method transforms the data to remove autocorrelation. A <strong>partial adjustment</strong> model lets consumers respond slowly: only about 28% of the adjustment happens within the first year, the short-run elasticity is &#8722;0.263 and the long-run elasticity is about &#8722;0.93 (the short-run elasticity divided by the speed of adjustment). Finally, because the gasoline price and GDP are non-stationary but share a long-run equilibrium with sales, an <strong>error correction</strong> model shows that if sales deviate from the long-run demand curve, about two-thirds of the gap closes within the first year.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FfBK62%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/faf64c18-c9ec-412c-8a58-dad597807502_1220x1128.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8b3ab3e8-7aff-4064-8528-26a3d7bfab41_1220x1248.png&quot;,&quot;height&quot;:535,&quot;title&quot;:&quot;Table 1. Common problems when estimating effects from time series&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FfBK62%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="535" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>The last row of Table 1 is the hardest. A higher price reduces purchases, but rising demand may also lead sellers to raise prices; the observed data are then the intersection of supply and demand, not the demand curve. Unless this endogeneity is addressed, every elasticity estimated above may be biased, however good the fit.</p><h2>3. What is Granger causality?</h2><p>The gasoline example starts from a causal assumption: price affects sales. Often we have no such assumption, as with the marketing and profit question at the start of this article. In 1969, the economist Clive Granger (later a Nobel laureate in economics in 2003) proposed an approach based on temporal order (Granger, 1969). This section explains the idea.</p><p>Suppose we want to know whether the oil price "causes" movements in the gold price. Step one is to forecast the gold price using only its own past values. Step two is to add past values of the oil price to the model. If adding past oil prices improves the forecast of gold <strong>in a statistically significant way</strong>, we say the oil price <strong>Granger-causes</strong> the gold price. The idea is that a cause cannot come after its effect, so if the past of X contains information that helps forecast Y, X may be one of Y's causes.</p><p>What must be clear from the outset is that this is a notion about <strong>forecasting</strong>, not about intervention. Many authors call it "predictive causality" to avoid confusion. We also usually test both directions: if X Granger-causes Y and Y also Granger-causes X, there is bidirectional Granger causality. With two variables, the model used to test both directions at once is the <strong>vector autoregression</strong> (VAR), in which each variable is forecast from its own past and the past of the other.</p><h2>4. Six steps to test for Granger causality</h2><p>Haughton et al. (2026) illustrate the procedure with daily gold prices (dollars per ounce) and oil prices (dollars per barrel) from 13 January 2015 to 1 November 2024, covering 2,441 days with both prices available. Figure 1 summarizes the six steps, discussed in turn below.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YK6x!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png 424w, https://substackcdn.com/image/fetch/$s_!YK6x!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png 848w, https://substackcdn.com/image/fetch/$s_!YK6x!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png 1272w, https://substackcdn.com/image/fetch/$s_!YK6x!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!YK6x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Six sequential steps: plot the series, check stationarity, choose the lag length, estimate the VAR model, test for Granger causality, plot impulse response functions&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Six sequential steps: plot the series, check stationarity, choose the lag length, estimate the VAR model, test for Granger causality, plot impulse response functions" title="Six sequential steps: plot the series, check stationarity, choose the lag length, estimate the VAR model, test for Granger causality, plot impulse response functions" srcset="https://substackcdn.com/image/fetch/$s_!YK6x!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png 424w, https://substackcdn.com/image/fetch/$s_!YK6x!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png 848w, https://substackcdn.com/image/fetch/$s_!YK6x!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png 1272w, https://substackcdn.com/image/fetch/$s_!YK6x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e2b71a-70d9-4e51-a7ca-d2185540e543_2075x583.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 1. Six steps for testing Granger causality. Source: based on Haughton et al. (2026).</figcaption></figure></div><p><strong>Step 1, plot the series.</strong> The gold price trends upward (partly because it is not adjusted for inflation), while the oil price fluctuates sharply with a spike and then a fall. Looking at the charts reveals trends, breaks and outliers before any test is run.</p><p><strong>Step 2, check stationarity.</strong> The ADF test shows that neither price series is stationary, but their differences are. Differences of logarithms, that is, daily percentage changes, are also stationary and are used in the following steps.</p><p><strong>Step 3, choose the lag length.</strong> You must decide how many periods back the model looks. Information criteria balance fit against parsimony, but they do not always agree: here the AIC suggests four lags while the BIC suggests none. The authors use two lags for illustration. Conclusions about Granger causality can change with the number of lags, so this is a step where sensitivity should be checked.</p><p><strong>Step 4, estimate the VAR.</strong> The results show that a rise in the oil price is followed by a rise in the gold price the next day and a fall the day after, both significant at the 5% level. The gold price also affects the oil price with a lag of two trading days.</p><p><strong>Step 5, test for Granger causality.</strong> The test asks: do lagged values of the other variable improve the model? Both directions reject the null hypothesis of "no causality", so there is <strong>bidirectional</strong> Granger causality between oil and gold prices.</p><p><strong>Step 6, plot impulse response functions.</strong> Because a VAR is dynamic, a shock to one variable spills over to the other and then feeds back over several periods. An <strong>impulse response function</strong> shows how each variable responds over time to a unit shock in the other, with a 95% confidence band. This step is optional but shows how large the effect is and how long it lasts, information a p-value does not give.</p><p>The result can be drawn as a <strong>time-series causal diagram</strong>, as in Figure 2. Each variable appears at several points in time, and arrows only run from past to present. Haughton et al. (2026) remark that it might have been better to start rather than finish with this diagram, because it forces the analyst to think about structure before estimating.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zX6B!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png 424w, https://substackcdn.com/image/fetch/$s_!zX6B!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png 848w, https://substackcdn.com/image/fetch/$s_!zX6B!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png 1272w, https://substackcdn.com/image/fetch/$s_!zX6B!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!zX6B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Time-series causal diagram: two rows of nodes for oil price and gold price at times t minus 2, t minus 1 and t; arrows from oil at t minus 1 and t minus 2 to gold at t, from gold at t minus 2 to oil at t, and from each variable to itself in the next period&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Time-series causal diagram: two rows of nodes for oil price and gold price at times t minus 2, t minus 1 and t; arrows from oil at t minus 1 and t minus 2 to gold at t, from gold at t minus 2 to oil at t, and from each variable to itself in the next period" title="Time-series causal diagram: two rows of nodes for oil price and gold price at times t minus 2, t minus 1 and t; arrows from oil at t minus 1 and t minus 2 to gold at t, from gold at t minus 2 to oil at t, and from each variable to itself in the next period" srcset="https://substackcdn.com/image/fetch/$s_!zX6B!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png 424w, https://substackcdn.com/image/fetch/$s_!zX6B!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png 848w, https://substackcdn.com/image/fetch/$s_!zX6B!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png 1272w, https://substackcdn.com/image/fetch/$s_!zX6B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcdbc2c-27b3-424e-80db-21fc92a0a3e9_1610x799.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 2. A time-series causal diagram for oil and gold prices. Source: based on results in Haughton et al. (2026).</figcaption></figure></div><h2>5. Business applications</h2><p>Granger causality is used heavily in macroeconomics, for example to examine whether consumption leads GDP or the reverse. In business, applications are less common but by no means rare. This section reviews three.</p><h3>5.1. Marketing spending and profit</h3><p>Back to the opening question. Banerjee and Siddhanta studied personal care companies in India using quarterly data from 2001 to 2011 (as cited in Haughton et al., 2026). The logarithms of profit and of marketing spending were non-stationary, but their differences were stationary. The AIC chose a lag of nine quarters, which makes sense because marketing may lower profit in the short run but raise it a year or two later. The two series were <strong>cointegrated</strong>, sharing a long-run equilibrium, so the VAR was extended to a vector error correction model. The result: marketing spending Granger-caused profit, with a lag of about one and a half to two years, while profit did not Granger-cause marketing spending. For the companies in the sample, the answer leaned towards "marketing creates profit".</p><h3>5.2. Competitor reactions</h3><p>Haughton et al. (2026) describe a consulting project in the telecoms industry that used weekly advertising spend to see whether one corporation's spending was followed by its main competitor's, or vice versa. If the competitor typically raises advertising a few weeks after the company does, the company needs to factor that reaction into its budget, because the sales it wins may be clawed back. Any industry with intense advertising competition, such as telecoms, digital wallets, food delivery or consumer goods, has a similar structure, provided competitors' spending can be obtained from media measurement firms.</p><h3>5.3. Media mix modeling</h3><p>The most common use of time series in marketing is <strong>media mix modeling</strong> (MMM): measuring the effect of spending on each channel (TV, social media, search, out-of-home) on sales in order to reallocate the budget. The standard technique is regression on weekly or monthly data. Haughton et al. (2026) point out three frequent weaknesses. First, short data: figures older than about five years may reflect a different market, so samples are small unless regional data exist. Second, pressure to find a good fit and significant coefficients leads to overfitting and "p-hacking". Third, the lack of data on competitors' spending, which affects one's own sales. The general reminder is that regression only measures association, so MMM results should be checked against experiments, such as switching a channel off or up in randomly chosen regions.</p><h2>6. Why Granger causality is not causality</h2><p>The name "Granger causality" invites misunderstanding. This section explains the conditions needed for a credible conclusion and the cases where it misleads.</p><h3>6.1. Coming first is not causing</h3><p>The familiar example: greeting card sales rise sharply a few weeks before Christmas, so card sales Granger-cause Christmas. Nobody believes cards cause Christmas; both are set by the calendar, and cards come first because people <strong>anticipate</strong> Christmas. Anticipation is a major source of spurious Granger causality in economics and business: stock prices move before announced earnings because investors anticipate earnings; advertising rises before peak season because companies anticipate the peak. Similarly, a third variable that affects two series with different lags makes the faster-reacting series look as if it "causes" the slower one.</p><h3>6.2. Stringent conditions</h3><p>A Granger causality test is only convincing when a series of conditions holds (Haughton et al., 2026). Table 2 lists them and what happens when they are violated.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FOIkwT%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/31b46fbe-a3f9-4e35-bcec-ba62a7b8e5e2_1220x776.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/59885906-e2b3-4a67-b772-0ca1fe4df6cb_1220x846.png&quot;,&quot;height&quot;:370,&quot;title&quot;:&quot;Table 2. Conditions for a credible Granger causality test&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FOIkwT%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="370" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>The conditions in Table 2 rarely hold together. That is why Shojaie and Fox (2022), in their review of Granger causality, conclude that while the concept is limited and generally not informative about causal effects, it can yield useful insights into interactions among variables observed over time. That is also how it should be used in business: as an exploratory tool for forming hypotheses and choosing where to experiment, not as final evidence for an investment decision. Newer methods for causal inference on time series, such as causal discovery algorithms that account for lags, are developing quickly but rest on similar assumptions.</p><h2>Conclusion</h2><p>Time series are the data companies have most of and the data most likely to mislead. Two trending series will correlate even when unrelated, like Ohio beavers and Irish GDP, so the first steps are always to plot the series, test stationarity and deal with the trend. When a causal assumption already exists, such as price affecting sales, estimation must still handle inflation, autocorrelation, slow adjustment and especially endogeneity, which can bias every figure however good the fit.</p><p>Granger causality answers a narrower question: does the past of one series help forecast another? Six steps, from plotting to impulse responses, give a clear procedure, and applications such as marketing and profit or competitor reactions show it can be useful. But coming first is not causing; anticipation and omitted variables both generate spurious Granger causality. The wise use is to treat the result as a clue about where to experiment, then verify with a stronger design such as a geographic or time-based randomized test.</p><h2>References</h2><p>Granger, C. W. J. (1969). Investigating causal relations by econometric models and cross-spectral methods. <em>Econometrica, 37</em>(3), 424&#8211;438. <a href="https://doi.org/10.2307/1912791">https://doi.org/10.2307/1912791</a></p><p>Haughton, D., Haughton, J., &amp; Lo, V. S. Y. (2026). <em>Cause and effect business analytics and data science</em>. CRC Press. <a href="https://doi.org/10.1201/9780429172588">https://doi.org/10.1201/9780429172588</a></p><p>Shojaie, A., &amp; Fox, E. B. (2022). Granger causality: A review and recent advances. <em>Annual Review of Statistics and Its Application, 9</em>, 289&#8211;319. <a href="https://doi.org/10.1146/annurev-statistics-040120-010930">https://doi.org/10.1146/annurev-statistics-040120-010930</a></p><p>Yule, G. U. (1926). Why do we sometimes get nonsense-correlations between time-series? A study in sampling and the nature of time-series. <em>Journal of the Royal Statistical Society, 89</em>(1), 1&#8211;63. <a href="https://doi.org/10.2307/2341482">https://doi.org/10.2307/2341482</a></p>]]></content:encoded></item><item><title><![CDATA[Causal Diagrams in Data Analysis]]></title><description><![CDATA[How causal diagrams work: chains, forks and colliders, the back-door criterion, good and bad controls, and learning DAGs from data.]]></description><link>https://www.phuonglib.com/p/causal-diagrams-in-data-analysis</link><guid isPermaLink="false">https://www.phuonglib.com/p/causal-diagrams-in-data-analysis</guid><dc:creator><![CDATA[PHUONGLIB]]></dc:creator><pubDate>Tue, 17 Nov 2026 16:23:24 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/e88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cGwR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!cGwR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!cGwR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!cGwR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!cGwR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A sunny miniature city with one-way arrow streets, a figure closing a gate across a side alley while the main avenue glows orange.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A sunny miniature city with one-way arrow streets, a figure closing a gate across a side alley while the main avenue glows orange." title="A sunny miniature city with one-way arrow streets, a figure closing a gate across a side alley while the main avenue glows orange." srcset="https://substackcdn.com/image/fetch/$s_!cGwR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!cGwR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!cGwR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!cGwR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe88323ff-2181-4d01-baca-70ed420acbbf_2560x1440.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><blockquote><p><strong>Key takeaways</strong></p><ul><li><p>A causal diagram (DAG) writes down what every regression silently assumes: which variable affects which. Missing arrows are the strong assumptions.</p></li><li><p>Every diagram is built from three structures: chains, common causes and colliders. Controlling for a collider creates a spurious association instead of removing one.</p></li><li><p>The back-door criterion turns this into a rule for choosing control variables; "add every variable you have" can make an estimate worse.</p></li><li><p>Patterns of independence in data can sometimes reveal the direction of an arrow, and algorithms such as PC automate this, but only under strong assumptions.</p></li><li><p>The greatest value of a diagram is the conversation it forces before modeling: what do we really believe causes what?</p></li></ul></blockquote><div><hr></div><p>A pharmaceutical company wants to know how to split its marketing budget for a new antibiotic among channels: sales representatives visiting physicians, other contacts, advertising in medical journals, and free samples. The analysts have 71 months of data on new prescriptions and 11 marketing variables. The fastest route is a regression with new prescriptions as the outcome and all 11 variables as predictors. The result is puzzling: journal advertising has a <strong>negative</strong> coefficient, as if more advertising made physicians prescribe less, and four variables are not significant because the variables are highly correlated with one another. Haughton et al. (2026) remark that this approach "substitutes data for thought", so a hard-to-interpret result should come as no surprise.</p><p>What is missing is an explicit assumption about which variable affects which. A <strong>causal diagram</strong>, most commonly in the form of a directed acyclic graph (DAG), is the tool for putting that assumption on paper. This article introduces causal diagrams: how to read them, how to use them to choose control variables, when data can suggest the direction of an arrow, and how algorithms learn diagrams from data.</p><p>The article has eight parts. Section 1 explains what a causal diagram is. Section 2 presents the three building blocks of every diagram. Section 3 uses them to answer the most practical question: which variables should go into a model. Section 4 shows that data can sometimes speak to causal direction, and Section 5 introduces structure-learning algorithms. Section 6 returns to the pharmaceutical example. Section 7 proposes a process for using causal diagrams in a project, and Section 8 discusses limitations.</p><h2>1. What is a causal diagram?</h2><p>A causal diagram consists of <strong>nodes</strong> (variables) and <strong>arrows</strong> between them. An arrow from A to B means we assume A has a direct effect on B. "Directed" means each arrow points one way; "acyclic" means you cannot follow the arrows back to where you started, since no variable can be its own cause. To describe feedback such as price affecting sales and sales affecting next period's price, we split variables by time (price in month 1, sales in month 1, price in month 2) so the diagram stays acyclic.</p><p>The most important thing to understand about causal diagrams is that <strong>the absence of an arrow</strong> is the strong assumption. Drawing an arrow from A to B only says there may be an effect, large or small, for the data to decide. Drawing no arrow between A and B asserts that A has no direct effect on B, and that assertion shapes the entire analysis that follows.</p><p>Causal diagrams are closely associated with Judea Pearl, who received the 2011 Turing Award for foundational contributions to probabilistic and causal reasoning in artificial intelligence. Pearl also coined the term <strong>Bayesian network</strong> for this kind of graph; a Bayesian network becomes a causal network when its arrows are read as causal relations rather than merely statistical dependence. <em>The Book of Why</em> (Pearl &amp; Mackenzie, 2018) is the most readable entry point for newcomers.</p><h2>2. Three building blocks</h2><p>Every causal diagram, however complex, is assembled from three basic three-variable structures. Understanding how information "flows" through them is enough to read most diagrams in practice. Figure 1 illustrates the three structures with business examples.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XE07!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png 424w, https://substackcdn.com/image/fetch/$s_!XE07!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png 848w, https://substackcdn.com/image/fetch/$s_!XE07!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png 1272w, https://substackcdn.com/image/fetch/$s_!XE07!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!XE07!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Three basic causal structures: a chain from advertising to website visits to sales; a common cause where the holiday season drives both promotions and sales; a collider where skills and connections both lead to being hired&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Three basic causal structures: a chain from advertising to website visits to sales; a common cause where the holiday season drives both promotions and sales; a collider where skills and connections both lead to being hired" title="Three basic causal structures: a chain from advertising to website visits to sales; a common cause where the holiday season drives both promotions and sales; a collider where skills and connections both lead to being hired" srcset="https://substackcdn.com/image/fetch/$s_!XE07!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png 424w, https://substackcdn.com/image/fetch/$s_!XE07!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png 848w, https://substackcdn.com/image/fetch/$s_!XE07!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png 1272w, https://substackcdn.com/image/fetch/$s_!XE07!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64b1708-a65d-4ebe-a638-e955f9d3aa41_1997x768.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 1. The three building blocks of causal diagrams.</figcaption></figure></div><p><strong>Chain</strong> (A &#8594; C &#8594; B): advertising increases website visits, and visits increase sales. Advertising and sales are correlated. But among days with the same number of visits, that is, <strong>holding</strong> (conditioning on) C fixed, advertising no longer relates to sales, because its whole effect runs through C. C is a <strong>mediator</strong>.</p><p><strong>Common cause</strong> (A &#8592; C &#8594; B): the holiday season makes the company run more promotions and also makes customers buy more. Promotions and sales are correlated even if promotions do nothing. Holding C fixed (comparing within the same season) removes the spurious association. C is a <strong>confounder</strong>.</p><p><strong>Common effect</strong> (A &#8594; C &#8592; B): skills and personal connections both help a candidate get hired. In the population, the two may be independent. But looking only at people who were hired, that is, holding C fixed, we see a negative association: hires with weaker skills tend to have better connections, and vice versa. C is a <strong>collider</strong>, and conditioning on it <strong>creates</strong> a spurious association rather than removing one.</p><p>Table 1 summarizes how the three structures behave. The first two behave alike statistically; the third behaves the opposite way, and that is the source of many mistakes.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FRbDnY%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3726bb1c-7b42-498e-81c5-2219e6dfa449_1220x584.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ce7e7cf7-256c-4054-83fe-468831442859_1220x704.png&quot;,&quot;height&quot;:319,&quot;title&quot;:&quot;Table 1. The three building blocks and what happens when the middle variable is controlled&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FRbDnY%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="319" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h2>3. Choosing control variables with a diagram</h2><p>The most practical question a causal diagram answers is: when estimating the effect of X on Y with a regression, which variables should be included? The habit of "the more variables the better" fails because it does not distinguish the three structures. This section presents the rule for choosing variables and the variables that should be left out.</p><h3>3.1. Back-door paths</h3><p>Between X and Y there may be several paths in the diagram. The path that follows the arrows from X to Y is the <strong>causal</strong> path, the thing we want to measure. Paths that begin with an arrow <strong>into</strong> X are called <strong>back-door paths</strong>, because they sneak into X from behind through a common cause. A small bubble tea shop estimating price sensitivity has the back-door path price &#8592; opening day &#8594; sales: the shop opened with low prices and heavy promotion, and both pushed sales up. If this path is left open, the price coefficient absorbs the effect of the opening.</p><p>Pearl's <strong>back-door criterion</strong> says that to estimate the effect of X on Y correctly, we must control for a set of variables that blocks every back-door path, and that set must contain no variable that is a consequence of X. A path is blocked if it passes through a controlled confounder or mediator, or through a collider that is <strong>not</strong> controlled (Pearl et al., 2016).</p><h3>3.2. Good and bad controls</h3><p>The back-door criterion yields practical rules about which variables to control. Cinelli et al. (2024) call this "a crash course in good and bad controls". Table 2 summarizes cases common in business analysis.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FrtBQK%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f0993356-0ed8-4737-85cd-f609dab6663e_1220x1032.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b67f70ff-ec26-4232-b1e4-dd6d9e757287_1220x1152.png&quot;,&quot;height&quot;:502,&quot;title&quot;:&quot;Table 2. Good and bad controls when estimating the effect of X on Y&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FrtBQK%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="502" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Table 2 explains a common phenomenon: adding variables to a model can make the coefficient of interest <strong>worse</strong>, not better. Controlling for "promoted" when estimating the effect of training creates a spurious association between training and productivity; controlling for "skills after the course" erases the very effect we want to measure.</p><h3>3.3. One model, one question</h3><p>A further consequence is that each regression model answers exactly one causal question. In the backpack campaign example of Haughton et al. (2026), gender affects both who the campaign reaches and the demand for backpacks. A model with both campaign and gender correctly estimates the campaign's effect, but the gender coefficient in the same model is only the direct effect of gender, excluding the part that runs through the campaign. Reading every coefficient in a regression table as a total effect is common enough that epidemiologists have named it the "Table 2 fallacy" (Westreich &amp; Greenland, 2013). Haughton et al. (2026) put it simply: regression coefficients have no causal meaning without an explicitly stated causal structure.</p><h2>4. Can data tell us the direction of an arrow?</h2><p>So far the diagram has been drawn from domain knowledge. A natural question is whether data can help determine the direction of an arrow. The answer is yes, in some cases, and the reason lies in the collider structure.</p><h3>4.1. A third variable reveals the direction</h3><p>With only two correlated variables A and B, data alone cannot tell whether A causes B or B causes A. But with a third variable C, the pattern of independence among the three can rule some possibilities out. Haughton et al. (2026) give an example: thin people tend to drink more diet soda. Are they thin because they drink diet soda, or do they drink it because they are thin? Suppose we also have age. If age is related to diet soda consumption (younger people drink more) but <strong>not</strong> to thinness, then diet soda cannot be what makes people thin; if it did, age would be indirectly related to thinness through diet soda. This logic is exactly the behavior of the collider and chain structures in Table 1.</p><p>A real example cited in the book is a study by Bryant and colleagues of alcohol and road deaths in the United States from 1947 to 1993 (as cited in Haughton et al., 2026). Alcohol consumption per head is correlated with road fatalities (coefficient 0.341, significant). Does alcohol cause accidents, or do accidents drive people to drink? The authors also had average highway speed: speed is correlated with deaths (0.386, significant) but not with alcohol consumption (0.136, not significant). As Figure 2 shows, the only consistent way to arrange these three relationships is that alcohol and speed both lead to deaths, making deaths a collider, so the causal direction runs from alcohol to deaths.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!a9VA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png 424w, https://substackcdn.com/image/fetch/$s_!a9VA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png 848w, https://substackcdn.com/image/fetch/$s_!a9VA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png 1272w, https://substackcdn.com/image/fetch/$s_!a9VA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!a9VA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Three-node diagram: alcohol consumption and highway speed both have arrows into road deaths; correlations 0.341 and 0.386 on the two arrows, 0.136 not significant between alcohol and speed&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Three-node diagram: alcohol consumption and highway speed both have arrows into road deaths; correlations 0.341 and 0.386 on the two arrows, 0.136 not significant between alcohol and speed" title="Three-node diagram: alcohol consumption and highway speed both have arrows into road deaths; correlations 0.341 and 0.386 on the two arrows, 0.136 not significant between alcohol and speed" srcset="https://substackcdn.com/image/fetch/$s_!a9VA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png 424w, https://substackcdn.com/image/fetch/$s_!a9VA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png 848w, https://substackcdn.com/image/fetch/$s_!a9VA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png 1272w, https://substackcdn.com/image/fetch/$s_!a9VA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36468ba6-42f6-4273-a8d0-d7d121311fe6_1455x860.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 2. A causal diagram of road deaths inferred from independence patterns. Source: based on Bryant and colleagues, as cited in Haughton et al. (2026).</figcaption></figure></div><h3>4.2. Three conditions for trusting the result</h3><p>This reasoning holds only when three conditions are met (Spirtes et al., 2000). The first is <strong>causal sufficiency</strong>: no omitted variable causes two of the variables in the model. Taken literally this can never be guaranteed, so in practice the analyst must argue convincingly that nothing important has been left out. The second is the <strong>causal Markov condition</strong>: each variable depends only on its direct causes (its "parents"), with no need to know about "grandparents" or "siblings". The third is <strong>faithfulness</strong>: two variables are dependent if and only if there is a path connecting them, so no two opposing effects cancel out so perfectly that the data look independent.</p><h2>5. Learning diagrams from data</h2><p>With dozens of variables, reasoning as in Section 4 by hand is impossible. Structure-learning algorithms do it systematically. This section introduces the idea behind the most popular algorithm, the available software and an example.</p><h3>5.1. The PC algorithm</h3><p>The PC algorithm (named after its authors Peter Spirtes and Clark Glymour) belongs to the family of <strong>constraint-based</strong> algorithms. Step one: connect every pair of variables with an undirected edge. Step two: remove edges between pairs that are independent, or independent conditional on some other set of variables, using correlation tests (continuous variables) or contingency-table tests (categorical variables). Step three: orient the remaining edges, starting with colliders. If X is linked to Y, Y to Z, but X and Z are independent, then X &#8594; Y &#8592; Z. From the identified colliders, the algorithm infers the direction of further edges (Spirtes et al., 2000).</p><p>A second family is <strong>score-based</strong> algorithms: search the space of possible diagrams for the one that fits the data best under a criterion such as BIC. This is a hard optimization problem because the number of diagrams explodes with the number of variables. Different algorithms often produce different diagrams, and finding good algorithms remains an active research field (Haughton et al., 2026).</p><h3>5.2. Software and a churn example</h3><p>The main software packages are Tetrad (Carnegie Mellon University, free), GeNIe, the bnlearn package in R, and BayesiaLab (commercial). In Python there is the causal-learn library. Haughton et al. (2026) ran the PC algorithm on data from a wireless provider wanting to know what makes customers switch. The data included many perception variables (reliability, innovativeness, sense of community), demographic variables (income, household size) and service-quality variables. The Tetrad result was surprising: the likelihood of switching was directly influenced only by <strong>satisfaction</strong>, and no perception variable had an arrow straight to churn. Rerunning in GeNIe, after imposing obvious constraints such as "age may affect spending, but spending cannot change age", added one more cause: the likelihood of the household acquiring more phones.</p><p>The example shows two things. The result does not immediately point to a churn solution, but it is a useful reality check on the common belief that brand image directly retains customers. And feeding background knowledge into the algorithm (which variables are certainly exogenous, which arrows are impossible) makes the diagram more sensible and useful. BayesiaLab also implements Pearl's "do-operator", which simulates what happens when a company <strong>actively changes</strong> a variable, for example increasing spending on one marketing channel, as opposed to merely observing that variable increase.</p><h2>6. Back to the pharmaceutical company</h2><p>Return to the opening example. The dataset in the book was simulated from the correlation matrix and summary statistics of a real study of a marketing campaign for a new antibiotic, because the original data were not published (Haughton et al., 2026). The regression with all 11 variables produced a negative coefficient for journal advertising and many insignificant variables, partly because the marketing variables are very highly correlated: months with many rep visits also have many samples and high contact costs.</p><p>When a causal diagram was built from the data and from knowledge of the industry, the picture became clearer: journal advertising most likely <strong>followed</strong> rather than led the growth in new prescriptions. In other words, the company advertised more once the drug was already selling well, rather than advertising being the main driver of sales. Haughton et al. (2026) suggest this finding might lead the marketing department to rethink how it spends its budget. It is also an example of reverse causation: a regression coefficient cannot tell the direction of an arrow, but a diagram forces the analyst to ask about it.</p><h2>7. A process for using causal diagrams in a project</h2><p>Causal diagrams need no expensive software or complex algorithms to get started. Much of their value comes from drawing them together with people who understand the business. Table 3 proposes a six-step process for an analytics project with a causal question.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FVOHrG%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bc7bbc96-d700-4f1a-87bb-1ee677c121c1_1220x1192.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2bc6e581-9828-41e4-8223-8fd54214b7c3_1220x1338.png&quot;,&quot;height&quot;:563,&quot;title&quot;:&quot;Table 3. Six steps for using a causal diagram in an analytics project&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FVOHrG%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="563" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Step 2 in Table 3 is often the most valuable. When marketing, sales and data teams draw the diagram together, hidden disagreements about "what causes what" surface before anyone runs a model, and many later arguments about numbers are settled right there. The free online tool DAGitty lets you draw a diagram and automatically identifies the adjustment set. When the diagram is fully specified in advance, the estimation step can use a structural equation model.</p><h2>8. Limitations</h2><p>A causal diagram is not magic that turns observational data into an experiment. It only makes assumptions transparent, and conclusions are only as good as the assumptions. If an unmeasured confounder affects both X and Y, no choice of controls can rescue the estimate, and designs such as instrumental variables or regression discontinuity become necessary.</p><p>Structure-learning algorithms have their own limits: results depend on the algorithm, the test thresholds and how variables are discretized, and many edges cannot be oriented from data at all. Haughton et al. (2026) stress that getting diagrams to "speak" involves some art and cannot be done mechanically. They also note that causal diagrams are still rarely used among business analysts, perhaps because regression is so quick and easy to run. That is a pity, because with a moderate number of variables, diagrams are easy to draw and read and can prevent investment decisions based on a coefficient with the wrong sign.</p><h2>Conclusion</h2><p>A causal diagram writes down what every regression implicitly assumes: which variable affects which. Three building blocks (chain, common cause, common effect) are enough to see why controlling for a confounder is right while controlling for a mediator or collider is wrong. The back-door criterion turns that insight into a rule for choosing control variables, replacing the habit of throwing every available variable into the model.</p><p>In some cases, data can even suggest the direction of an arrow through patterns of independence, as in the example of alcohol, speed and road deaths; algorithms such as PC automate that reasoning for many variables. But diagrams learned from data need checking by people who know the business, and every conclusion still depends on the assumption that no important variable has been left out. The greatest value of a causal diagram may lie not in the algorithm but in the conversation it forces a team to have before running a model: what do we really believe causes what?</p><h2>References</h2><p>Cinelli, C., Forney, A., &amp; Pearl, J. (2024). A crash course in good and bad controls. <em>Sociological Methods &amp; Research, 53</em>(3), 1071&#8211;1104. <a href="https://doi.org/10.1177/00491241221099552">https://doi.org/10.1177/00491241221099552</a></p><p>Haughton, D., Haughton, J., &amp; Lo, V. S. Y. (2026). <em>Cause and effect business analytics and data science</em>. CRC Press. <a href="https://doi.org/10.1201/9780429172588">https://doi.org/10.1201/9780429172588</a></p><p>Pearl, J., Glymour, M., &amp; Jewell, N. P. (2016). <em>Causal inference in statistics: A primer</em>. Wiley. <a href="https://www.wiley.com/en-us/Causal+Inference+in+Statistics:+A+Primer-p-9781119186847">https://www.wiley.com/en-us/Causal+Inference+in+Statistics:+A+Primer-p-9781119186847</a></p><p>Pearl, J., &amp; Mackenzie, D. (2018). <em>The book of why: The new science of cause and effect</em>. Basic Books. <a href="https://www.hachettebookgroup.com/titles/judea-pearl/the-book-of-why/9781541608986/">https://www.hachettebookgroup.com/titles/judea-pearl/the-book-of-why/9781541608986/</a></p><p>Spirtes, P., Glymour, C., &amp; Scheines, R. (2000). <em>Causation, prediction, and search</em> (2nd ed.). MIT Press. <a href="https://mitpress.mit.edu/9780262194402/causation-prediction-and-search/">https://mitpress.mit.edu/9780262194402/causation-prediction-and-search/</a></p><p>Westreich, D., &amp; Greenland, S. (2013). The Table 2 fallacy: Presenting and interpreting confounder and modifier coefficients. <em>American Journal of Epidemiology, 177</em>(4), 292&#8211;298. <a href="https://doi.org/10.1093/aje/kws412">https://doi.org/10.1093/aje/kws412</a></p>]]></content:encoded></item><item><title><![CDATA[From Business Problem to Data Solution: The Machine Learning Canvas]]></title><description><![CDATA[Four questions to find the real business problem, two traps in measuring success, and the Machine Learning Canvas applied to four types of project.]]></description><link>https://www.phuonglib.com/p/machine-learning-canvas-business-problem</link><guid isPermaLink="false">https://www.phuonglib.com/p/machine-learning-canvas-business-problem</guid><dc:creator><![CDATA[PHUONGLIB]]></dc:creator><pubDate>Tue, 17 Nov 2026 16:23:22 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Zojj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!Zojj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!Zojj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!Zojj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Zojj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An architect holding an orange translucent blueprint to the sky, through which a pile of loose bricks appears as a finished house.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An architect holding an orange translucent blueprint to the sky, through which a pile of loose bricks appears as a finished house." title="An architect holding an orange translucent blueprint to the sky, through which a pile of loose bricks appears as a finished house." srcset="https://substackcdn.com/image/fetch/$s_!Zojj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!Zojj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!Zojj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!Zojj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e172541-494d-47c8-9018-f62771eafc8d_2560x1440.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><blockquote><p><strong>Key takeaways</strong></p><ul><li><p>A 95%-accurate model can still create no business value, usually because the team skipped <strong>business understanding</strong> and solved a surface request instead of the real problem.</p></li><li><p>Four questions close the gap: What is the real problem and why does it matter? Who is the end user? How will the solution fit into the workflow? How will success be measured?</p></li><li><p>Two traps await when measuring success: accuracy is meaningless on imbalanced data, and predicting who will churn is not the same as knowing who can be kept (uplift); a model's value lies in the extra customers it creates over a control group.</p></li><li><p>Louis Dorard's Machine Learning Canvas puts every answer on one page in ten blocks across Learn, Predict and Evaluate. Four worked examples show it for classification, regression, clustering and recommendation.</p></li><li><p>When several requests compete, score each on value and feasibility using the canvas blocks; the matrix separates strategic bets, quick wins and projects that need foundations first.</p></li><li><p>The canvas complements CRISP-DM: CRISP-DM describes the sequence of work, while the canvas records the design decisions.</p></li></ul></blockquote><div><hr></div><p>In data science, we are often captivated by the complexity of algorithms, the glamour of "state-of-the-art" models and the thrill of squeezing out another percentage point of accuracy. We rush to collect data, preprocess it, build a model and deploy it. Then a sad truth emerges: a model with 95% accuracy brings the business no value, or worse, nobody uses it.</p><p>This familiar story comes from skipping a quiet but critical foundation: <strong>business understanding</strong>. This stage is not about code or algorithms but about understanding the business problem deeply. Before writing the first line of code, you must be able to answer: "What problem are we really trying to solve?" Skimping on this stage leads to wasted resources, products nobody adopts and stakeholders who lose faith in data science.</p><p>The article has seven parts. Section 1 describes the gap between a "surface request" and the "real problem". Section 2 sets out four core questions that close it, each with two examples. Section 3 examines two common traps in measuring success, including the gap between response models and uplift models. Section 4 introduces the Machine Learning Canvas, a tool that captures all the answers on one page, and Section 5 applies it to four types of problem: classification, regression, clustering and recommendation. Section 6 prioritises when several problems arrive at once, using a value and feasibility matrix. Section 7 places business understanding within the CRISP-DM cycle.</p><h2>1. The gap between request and problem</h2><p>Most data projects begin with a short sentence from the business: "We need a customer churn prediction model," or "Build us a recommendation system." It sounds like a clear technical requirement but is only a <strong>surface request</strong>. It does not say who will use the result, for what, when, or what counts as success.</p><p>Business understanding requires the data professional to act as a consultant: to listen and ask the right questions to move from the surface request to the <strong>real problem</strong>. Figure 1 summarises that journey through the four core questions detailed in Section 2.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zIas!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png 424w, https://substackcdn.com/image/fetch/$s_!zIas!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png 848w, https://substackcdn.com/image/fetch/$s_!zIas!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png 1272w, https://substackcdn.com/image/fetch/$s_!zIas!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!zIas!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Diagram: the surface request \&quot;build a churn prediction model\&quot; passes through four questions to become the real problem: reduce churn among high-value customers, measured by an A/B test&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Diagram: the surface request &quot;build a churn prediction model&quot; passes through four questions to become the real problem: reduce churn among high-value customers, measured by an A/B test" title="Diagram: the surface request &quot;build a churn prediction model&quot; passes through four questions to become the real problem: reduce churn among high-value customers, measured by an A/B test" srcset="https://substackcdn.com/image/fetch/$s_!zIas!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png 424w, https://substackcdn.com/image/fetch/$s_!zIas!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png 848w, https://substackcdn.com/image/fetch/$s_!zIas!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png 1272w, https://substackcdn.com/image/fetch/$s_!zIas!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4535351e-1cff-4c02-8e14-cc20efd42d88_1920x768.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 1. Four questions turn a surface request into the real problem.</figcaption></figure></div><h2>2. Four core questions for business understanding</h2><p>Each question below is illustrated with two running examples, customer churn prediction and product recommendation, each in three steps: the surface request, the probing questions and the real problem uncovered.</p><h3>2.1. What is the real problem, and why does it matter?</h3><p>This is the most fundamental question. Never accept the initial request mechanically. Dig for the business's real "pain" and why solving it is urgent.</p><p><strong>Example 1: churn prediction.</strong> Surface request: "We need a churn prediction model." Probing questions: "Why does predicting churn matter now? How much revenue is churn actually costing? Is the end goal just to know who will leave, or to <em>keep</em> them?" Real problem (illustrative figures): the company is losing 15% of revenue each quarter as high-value customers leave. The goal is not "prediction" but <strong>cutting churn among VIP customers to 5%</strong> by detecting it early and offering suitable care.</p><p><strong>Example 2: product recommendation.</strong> Surface request: "Build a product recommendation system to increase revenue." Probing questions: "How do we want to increase revenue? Up-selling more expensive products, or cross-selling related ones? Where will recommendations appear on the site?" Real problem (illustrative figures): customers usually buy one product and leave. The goal is to <strong>raise average order value (AOV) by 10%</strong> by suggesting products frequently bought together, right on the cart page.</p><h3>2.2. Who will be the end user?</h3><p>However perfect a technical solution is, it fails if users cannot or will not use it.</p><p><strong>Example 1.</strong> Surface request: "Just export an Excel file of customers at risk of churning." Probing questions: "Who will read this file? Marketing or customer care? What is their current workflow? What will they do with the list?" Real problem: the <strong>customer care team</strong> needs the list built straight into their CRM. More importantly, they need to know <strong>why</strong> a customer is flagged, such as "usage down 50%" or "called support three times this week", to personalise the retention call.</p><p><strong>Example 2.</strong> Surface request: "Show recommended products." Probing questions: "Who sees these recommendations? At what stage of the shopping journey? Are they in a hurry?" Real problem: the <strong>shopper</strong> is the end user, on the cart page and about to pay. Recommendations must be fast, truly relevant and non-distracting; otherwise they may abandon the cart altogether.</p><h3>2.3. How will the solution be integrated and used?</h3><p>This question determines the technical architecture and how the final product operates.</p><p><strong>Example 1.</strong> Surface request: "We need a model." Probing questions: "How often must results be updated, daily or weekly? How will they reach users?" Real problem: a <strong>batch-processing system</strong> that runs at the start of each month, scans all customers and pushes the highest-risk list into the CRM through an API.</p><p><strong>Example 2.</strong> Surface request: "A recommendation system." Probing questions: "Must predictions happen the moment a user adds an item to the cart? What latency is acceptable?" Real problem: a <strong>real-time API</strong> with latency under 100 milliseconds that takes the cart's products as input and returns a ranked list of recommended products.</p><h3>2.4. How will we measure success?</h3><p>This is where we move from <strong>model metrics</strong> to <strong>business metrics</strong>.</p><p><strong>Example 1.</strong> Surface request: "The model needs over 90% accuracy." Probing questions: "What if the model reaches 90% accuracy but churn does not fall? What does a wrong prediction cost, for example flagging a loyal customer as about to leave?" Real problem: success is measured by an <strong>A/B test</strong>: actual churn among customers who receive model-driven intervention must be <strong>at least 15% lower</strong> than in a control group with no intervention.</p><p><strong>Example 2.</strong> Surface request: "Click-through on recommendations must be high." Probing questions: "What if users click a lot but don't buy? Isn't the end goal more revenue?" Real problem: success is measured by an <strong>A/B test</strong>: <strong>average order value</strong> for customers who see recommendations must be <strong>at least 10% higher</strong> than for those who don't.</p><p>Table 1 summarises the four questions and the real problems found in the two examples.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F7dB1L%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c276138f-19ce-4136-a669-86586389e854_1220x660.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e8d617e2-4799-4c43-883f-1b8ca0df1716_1220x818.png&quot;,&quot;height&quot;:347,&quot;title&quot;:&quot;Table 1. Four core questions and the real problem in two examples&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F7dB1L%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="347" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>The fourth question is where projects stumble most, because two traps are very easy to fall into. The next section analyses them.</p><h2>3. Two traps in measuring success</h2><p>Once the business problem has become a data problem, the next step is choosing a measure of success. Many projects go wrong here without realising it. The two common traps are trusting a high accuracy figure on imbalanced data, and confusing predicting who will churn with knowing who can be kept.</p><h3>3.1. Why "90% accuracy" can be meaningless</h3><p>Return to the request "the model needs over 90% accuracy". Suppose the company has 10,000 customers and only 5%, or 500, churn each month. Figure 2 compares two models (illustrative figures).</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CPvV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png 424w, https://substackcdn.com/image/fetch/$s_!CPvV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png 848w, https://substackcdn.com/image/fetch/$s_!CPvV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png 1272w, https://substackcdn.com/image/fetch/$s_!CPvV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!CPvV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Bar chart: a model that always answers \&quot;no churn\&quot; has 95% accuracy but zero precision and recall; a model flagging 1,000 customers has 91% accuracy, 30% precision and 60% recall&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Bar chart: a model that always answers &quot;no churn&quot; has 95% accuracy but zero precision and recall; a model flagging 1,000 customers has 91% accuracy, 30% precision and 60% recall" title="Bar chart: a model that always answers &quot;no churn&quot; has 95% accuracy but zero precision and recall; a model flagging 1,000 customers has 91% accuracy, 30% precision and 60% recall" srcset="https://substackcdn.com/image/fetch/$s_!CPvV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png 424w, https://substackcdn.com/image/fetch/$s_!CPvV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png 848w, https://substackcdn.com/image/fetch/$s_!CPvV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png 1272w, https://substackcdn.com/image/fetch/$s_!CPvV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89879f37-0e95-46c5-8fc2-3042e7cd7f74_1917x912.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 2. The accuracy paradox on imbalanced data. Illustrative figures.</figcaption></figure></div><p><strong>Model A</strong> is a lazy "model" that always answers "no churn". It is right for 9,500 of 10,000 customers, 95% accuracy, above the requirement. But it finds not a single customer to retain. <strong>Model B</strong> flags 1,000 customers, of whom 300 really churn. Its accuracy is only 91%, lower than A's. But it finds 60% of the customers about to leave.</p><p>With imbalanced data like this, look at <strong>precision</strong> (of those flagged, how many really churn: 30% for model B) and <strong>recall</strong> (of those who churn, how many the model finds: 60%). Then convert them into money: how much a retention call costs and how much losing a customer costs. If each call costs about USD 2 and each retained customer is worth about USD 80, model B's 1,000 calls cost USD 2,000, and keeping just 25 of the 300 customers about to leave breaks even (illustrative figures).</p><h3>3.2. Predicting who will churn is not predicting who can be kept</h3><p>The second trap is subtler. Even a very accurate churn model can waste the budget because it answers the wrong question. Figure 3 divides customers into four groups by how they respond to being contacted (Radcliffe &amp; Surry, 2011).</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dHaP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png 424w, https://substackcdn.com/image/fetch/$s_!dHaP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png 848w, https://substackcdn.com/image/fetch/$s_!dHaP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png 1272w, https://substackcdn.com/image/fetch/$s_!dHaP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!dHaP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Matrix of four customer groups: sure things, sleeping dogs, persuadables and lost causes&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Matrix of four customer groups: sure things, sleeping dogs, persuadables and lost causes" title="Matrix of four customer groups: sure things, sleeping dogs, persuadables and lost causes" srcset="https://substackcdn.com/image/fetch/$s_!dHaP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png 424w, https://substackcdn.com/image/fetch/$s_!dHaP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png 848w, https://substackcdn.com/image/fetch/$s_!dHaP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png 1272w, https://substackcdn.com/image/fetch/$s_!dHaP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02c711a5-aaa5-4444-ac06-01eaaf3dda5f_1517x984.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 3. Four customer groups by response to a retention programme. Source: adapted from Radcliffe and Surry (2011).</figcaption></figure></div><p>Some customers are <strong>sure things</strong>: calling them is wasted money. Some are <strong>lost causes</strong>: they have already decided to switch provider, and no offer will keep them. Some are <strong>persuadables</strong>, who stay only if contacted: this is where the budget should go. And some are <strong>sleeping dogs</strong>: a call reminds them their contract is ending, prompting them to compare prices and leave.</p><p>A conventional churn model finds high-risk customers, lumping persuadables and lost causes together. <strong>Uplift models</strong> are built to find the persuadables specifically, and they need data from experiments with control groups. That is why question 4 almost always leads to an A/B test.</p><p>Haughton et al. (2026) tell a real story showing how this trap plays out inside an organisation. A large company had three teams: a modelling team that built response models to select customers, a marketing team that designed campaigns, and a measurement team that evaluated results. Customers in the model's top two deciles were randomly split into a group that received a mailed offer and a group that did not; a randomly chosen set of customers was split in the same way. The results are in Table 2: customers chosen by the model responded at 1.0%, more than three times the 0.3% of random customers, so the modelling team declared the model a success. But the measurement team pointed out that among model-selected customers, those who received the mail and those who did not both responded at 1.0%: the campaign created no extra customers at all. The three teams blamed one another until a new modelling manager saw that the problem lay in the model's design. It had been built to find people who <strong>would buy</strong>, whereas the business needed people who <strong>buy because they were contacted</strong>.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FwzVzt%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e4d9dabb-3aaa-48e6-bbf0-d42c5e3d506d_1220x490.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/73b4e3e2-6b9c-4cb6-bc42-2bdf58b7ee7d_1220x610.png&quot;,&quot;height&quot;:258,&quot;title&quot;:&quot;Table 2. Two ways of reading the same campaign (response rate, %)&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FwzVzt%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="258" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Table 2 shows why the right metric matters. Lift over random (bottom row, first column: 0.7 percentage points) proves only that the model finds people inclined to buy. The model's business value is a <strong>difference of differences</strong>: the effect of the offer among model-selected customers minus the effect of the offer among random customers (the bottom-right cell). That is the quantity uplift models are built to maximise, and the one that belongs in the answer to "how will we measure success?". Uplift matters most when the campaign's effect is small relative to natural purchasing: if 5% of customers buy anyway and the offer adds only 0.01 percentage points, a conventional response model learns almost nothing but natural buying behaviour. The best-known application is Barack Obama's 2012 re-election campaign: rather than contacting voters certain to vote for him or certain to vote for his opponent, the analytics team used uplift models to find the most persuadable voters and concentrated resources on them.</p><p>Asking questions is necessary, but the answers easily get lost in meeting notes and emails. A tool is needed to keep them on one page.</p><h2>4. A tool to systematise: the Machine Learning Canvas</h2><p>Inspired by the famous <strong>Business Model Canvas</strong> from the start-up world, Louis Dorard created the <strong><a href="https://www.ownml.co/machine-learning-canvas">Machine Learning Canvas</a></strong><a href="https://www.ownml.co/machine-learning-canvas"> (MLC)</a>, a visual one-page blueprint that helps all stakeholders, from business and product to engineering, look in the same direction, use a shared language and make sure no important aspect is overlooked (Dorard, n.d.). Figure 4 shows Dorard's canvas.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!IKb6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp 424w, https://substackcdn.com/image/fetch/$s_!IKb6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp 848w, https://substackcdn.com/image/fetch/$s_!IKb6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp 1272w, https://substackcdn.com/image/fetch/$s_!IKb6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!IKb6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Louis Dorard's Machine Learning Canvas with blocks for Value Proposition, Prediction Task, Decisions, Making Predictions, Offline Evaluation, Data Collection, Data Sources, Building Models, Features and Live Monitoring&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Louis Dorard's Machine Learning Canvas with blocks for Value Proposition, Prediction Task, Decisions, Making Predictions, Offline Evaluation, Data Collection, Data Sources, Building Models, Features and Live Monitoring" title="Louis Dorard's Machine Learning Canvas with blocks for Value Proposition, Prediction Task, Decisions, Making Predictions, Offline Evaluation, Data Collection, Data Sources, Building Models, Features and Live Monitoring" srcset="https://substackcdn.com/image/fetch/$s_!IKb6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp 424w, https://substackcdn.com/image/fetch/$s_!IKb6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp 848w, https://substackcdn.com/image/fetch/$s_!IKb6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp 1272w, https://substackcdn.com/image/fetch/$s_!IKb6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82e8095f-e8bf-4724-bc55-4c5f40ccceb3_1280x958.webp 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 4. The Machine Learning Canvas. Source: Louis Dorard, CC BY-SA licence.</figcaption></figure></div><p>The canvas has a central block and three sections, ten blocks in all. Table 3 summarises the question each block must answer; the subsections below explain them.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FXtDpt%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/65cdc687-c560-4626-9731-9ff73a69c58a_1220x1552.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/06d410af-ea2e-43c4-b3b0-5c24bb0428ee_1220x1622.png&quot;,&quot;height&quot;:782,&quot;title&quot;:&quot;Table 3. The ten blocks of the Machine Learning Canvas&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FXtDpt%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="782" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>4.1. The central block: value proposition</h3><p>This is the heart of the canvas, where the answers to the four questions in Section 2 crystallise. It forces you to state the problem, its importance and the end user clearly. An effective formula for writing it comes from Geoffrey Moore's <em>Crossing the Chasm</em> (Moore, 2014):</p><blockquote><p><strong>For</strong> (target customers) <strong>who</strong> (have a need or opportunity), (our product or service) <strong>is a</strong> (product category) <strong>that</strong> (delivers a benefit).</p></blockquote><h3>4.2. LEARN: the data and training foundation</h3><p>This section covers everything to do with data and how the model "learns".</p><p><strong>Data Sources</strong> lists every usable source: internal databases (transaction systems, data warehouses), third-party APIs, static files (CSV, Excel), web data, even the output of another machine-learning system. Naming sources clearly helps estimate the complexity of collection and integration.</p><p><strong>Collecting Data</strong> answers: how will the system keep learning and avoid going stale (model decay)? In particular, how will we get new <strong>labels</strong> for the output? Do people need to label manually (human in the loop), and at what cost?</p><p><strong>Features</strong> describes how raw data becomes signals the model understands. This is feature engineering: from transaction dates, we create days since last purchase, purchase frequency and so on. Consulting domain experts is especially valuable here.</p><p><strong>Building Models</strong> sets the cycle and constraints of retraining: how often (hourly, daily, weekly), how long it takes and hidden costs such as cloud resources.</p><h3>4.3. PREDICT: the core logic of the system</h3><p>This section describes how the system uses the model to create results and impact.</p><p><strong>Prediction Task</strong> defines the technical problem precisely. Despite its name, the block covers any machine-learning task: classification, regression, clustering, recommendation. Define the inputs, outputs and acceptable complexity.</p><p><strong>Decisions</strong> is the most important bridge between results and value. It answers "So what?": how does a model's output become a business <strong>action</strong>? For example, if an email is classified as spam, the system automatically moves it to the spam folder.</p><p><strong>Making Predictions</strong> sets when and how the system produces results for new data: every time a user opens the app or on a fixed schedule; for each data point (real time) or for a whole batch.</p><h3>4.4. EVALUATE: measurement and validation</h3><p>This section ensures the project's success can be measured before and after deployment.</p><p><strong>Offline Evaluation</strong> sets criteria for assessing the model in a test environment before it affects real users: technical metrics such as precision, recall and F1, and more importantly business metrics, such as simulating revenue if the model were applied to the test set (like the break-even calculation in Section 3.1).</p><p><strong>Live Evaluation &amp; Monitoring</strong> tracks real-world performance and measures the business value created, usually with an A/B test: has actual churn fallen?</p><p>Ten blocks sound abstract. The best way to understand them is to fill them in for real projects.</p><h2>5. Four worked examples</h2><p>The four examples below correspond to four common types of machine-learning problem. Each starts with a surface request and ends with a ten-block canvas.</p><h3>5.1. Classification: predicting customer churn</h3><p><strong>Situation:</strong> the leadership of a telecoms company says: "We're losing too many customers. Build a churn prediction model to fix this."</p><ul><li><p><strong>Value Proposition:</strong> "For the customer care team, who need to identify high-value subscribers at risk of leaving, our early-warning system is an analytics tool that helps them reach out proactively with suitable offers to reduce churn."</p></li><li><p><strong>Data Sources:</strong> CRM (customer information), billing database (charge history), customer service system (support calls), usage data (data volume, call minutes).</p></li><li><p><strong>Collecting Data:</strong> each month, the system automatically labels customers who do not renew or pay as "churned"; this data is used to retrain the model.</p></li><li><p><strong>Features:</strong> tenure, average bill over the last three months, days overdue on payment, number of support calls, percentage change in data usage versus the previous month.</p></li><li><p><strong>Building Models:</strong> retrained on the 1st of each month with the previous month's data, running overnight.</p></li><li><p><strong>Prediction Task:</strong> binary classification. Input: the customer's feature vector. Output: a "churn" or "no churn" label and a probability score.</p></li><li><p><strong>Decisions:</strong> when a customer's churn score exceeds 0.8, a high-priority task is created automatically in the CRM and assigned to a care agent, with the main reasons.</p></li><li><p><strong>Making Predictions:</strong> batch processing on the 1st of each month, scanning all current customers.</p></li><li><p><strong>Offline Evaluation:</strong> precision and recall on the test set, emphasising recall so high-risk customers are not missed while keeping precision high enough that agents don't waste time on customers with no intention of leaving; a simulation of retention cost versus revenue retained.</p></li><li><p><strong>Live Evaluation &amp; Monitoring:</strong> an A/B test comparing actual churn between the intervention group and a control group. Target (illustrative): 15% lower.</p></li></ul><p><strong>Result:</strong> a shift from "prediction" to an automated, measurable "retention action" process that can be continuously improved.</p><h3>5.2. Regression: predicting house prices</h3><p><strong>Situation:</strong> a property company wants a tool "to predict house prices for sales staff".</p><ul><li><p><strong>Value Proposition:</strong> "For property sales staff, who spend a lot of time setting asking prices, the price estimator is a feature in the internal software that gives them a reliable reference price instantly, speeding up quotes."</p></li><li><p><strong>Data Sources:</strong> internal data on completed sales, data from property listing sites, open data on zoning, schools and hospitals.</p></li><li><p><strong>Collecting Data:</strong> each week, the system automatically updates newly listed and sold properties.</p></li><li><p><strong>Features:</strong> floor area, bedrooms, bathrooms, district, distance to the nearest metro station, building age, parking.</p></li><li><p><strong>Building Models:</strong> retrained every Sunday for the new working week.</p></li><li><p><strong>Prediction Task:</strong> regression. Input: the property's feature vector. Output: an estimated value.</p></li><li><p><strong>Decisions:</strong> the model provides a "suggested price range" (for example, USD 190,000 to 210,000, illustrative), which staff use as the basis for the official asking price (human in the loop).</p></li><li><p><strong>Making Predictions:</strong> on demand: when staff enter a property's details, the system calls the model's API and returns the result at once.</p></li><li><p><strong>Offline Evaluation:</strong> mean absolute percentage error (MAPE); illustrative target under 10%.</p></li><li><p><strong>Live Evaluation &amp; Monitoring:</strong> average time from request to quote (expected to fall by 30%) and the gap between quoted and actual sale price.</p></li></ul><p><strong>Result:</strong> a "smart assistant" that does not replace people but helps sales staff work more efficiently. Leaving the final price to staff is a deliberate choice: the model doesn't know things you only learn by visiting, such as which way the house faces or what the neighbours are like.</p><h3>5.3. Clustering: customer segmentation</h3><p><strong>Situation:</strong> the marketing team of an e-commerce site proposes: "Segment our customers so we can market better."</p><ul><li><p><strong>Value Proposition:</strong> "For the marketing team, who currently send the same message to every customer, the automated segmentation system is a data source that helps them design personalised email and push campaigns to raise engagement and conversion."</p></li><li><p><strong>Data Sources:</strong> the order database, the customer database, web behaviour data.</p></li><li><p><strong>Collecting Data:</strong> behavioural and transaction data are collected continuously.</p></li><li><p><strong>Features:</strong> the RFM model: Recency (last purchase), Frequency (how often), Monetary (total spend).</p></li><li><p><strong>Building Models:</strong> rerun the clustering algorithm (such as K-Means) at the start of each week.</p></li><li><p><strong>Prediction Task:</strong> clustering. Input: each customer's RFM vector. Output: the customer's cluster, named for example "VIP", "loyal" or "dormant".</p></li><li><p><strong>Decisions:</strong> each segment gets a different campaign: discount codes for VIPs, a "We miss you!" email for dormant customers.</p></li><li><p><strong>Making Predictions:</strong> weekly batch processing, updating segment labels for all customers.</p></li><li><p><strong>Offline Evaluation:</strong> cluster-quality metrics such as the silhouette score; marketing experts review each cluster's characteristics to ensure they make business sense.</p></li><li><p><strong>Live Evaluation &amp; Monitoring:</strong> compare open, click and conversion rates of personalised campaigns with the previous one-size-fits-all campaigns.</p></li></ul><p><strong>Result:</strong> a vague request to "segment" becomes a data-driven marketing strategy with clear actions and metrics.</p><h3>5.4. Recommendation: cross-selling in the cart</h3><p><strong>Situation:</strong> an e-commerce company asks: "Build a product recommendation system to increase revenue."</p><ul><li><p><strong>Value Proposition:</strong> "For customers viewing their cart, the 'Frequently bought together' feature is a product list that helps them find useful items easily and get more value from this purchase."</p></li><li><p><strong>Data Sources:</strong> detailed history of successfully paid orders.</p></li><li><p><strong>Collecting Data:</strong> new order data is collected in real time.</p></li><li><p><strong>Features:</strong> no complex features needed; the input is the set of products in each cart.</p></li><li><p><strong>Building Models:</strong> rerun an association-rule mining algorithm (such as Apriori) weekly to find new rules (for example, illustratively: customers who buy milk have a 60% chance of buying nappies).</p></li><li><p><strong>Prediction Task:</strong> recommendation. Input: the products in the current cart. Output: a list of recommended products ranked by rule strength.</p></li><li><p><strong>Decisions:</strong> when a user adds product A and a sufficiently strong rule {A} &#8594; {B} exists, the system shows B under "Frequently bought together".</p></li><li><p><strong>Making Predictions:</strong> real time, looking up precomputed rules.</p></li><li><p><strong>Offline Evaluation:</strong> evaluate rules by support, confidence and lift.</p></li><li><p><strong>Live Evaluation &amp; Monitoring:</strong> an A/B test with average order value as the core metric.</p></li></ul><p><strong>Result:</strong> choosing the right technical method goes straight at the business goal (higher AOV) and avoids building an unnecessarily complex system.</p><p>Table 4 sets the four canvases side by side on the blocks where they differ most.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F5ysa4%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/faa8ccd3-69c0-4a2f-9557-aa6642b7a5ba_1220x894.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e43166a8-fbb1-4510-a7ca-bb89925649a1_1220x964.png&quot;,&quot;height&quot;:415,&quot;title&quot;:&quot;Table 4. The four examples compared on key blocks&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F5ysa4%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="415" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Table 4 shows one thing clearly: the "live evaluation" row is always a business metric, not a model metric. The canvas forces every project to answer question 4 from the start. But the canvas is only the first step of a larger cycle.</p><h2>6. When there are many problems: which to tackle first</h2><p>The four examples in Section 5 show how a canvas clarifies one project. In practice a data team rarely receives one request; it receives five or six at once, each with a sponsor who believes theirs matters most. This section uses the canvas blocks themselves to set priorities.</p><h3>6.1. Scoring value and feasibility</h3><p>Each problem is scored from 1 to 5 on five criteria in two groups. <strong>Value</strong> covers financial impact (weight 30%) and strategic fit (20%); these scores come from the Value Proposition and Decisions blocks. <strong>Feasibility</strong> covers data readiness (20%), technical ease (15%) and low legal or reputational risk (15%); these come from the Data Sources, Collecting Data and Features blocks. The value and feasibility scores are weighted averages within each group. Table 5 scores six illustrative requests received by the data team of a telecom operator in one quarter.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2Fd2RDF%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cd018fad-0fe9-4987-b9d3-f560fd5f434e_1220x936.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4a3027c1-1dde-4bc2-88ea-083afbca994a_1220x1056.png&quot;,&quot;height&quot;:528,&quot;title&quot;:&quot;Table 5. Scores for six requests at a telecom operator (scale 1 to 5, illustrative)&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2Fd2RDF%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="528" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Figure 5 places the six requests from Table 5 on a matrix with a value threshold of 3.5 and a feasibility threshold of 3.0.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!VtAX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png 424w, https://substackcdn.com/image/fetch/$s_!VtAX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png 848w, https://substackcdn.com/image/fetch/$s_!VtAX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png 1272w, https://substackcdn.com/image/fetch/$s_!VtAX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!VtAX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Value and feasibility matrix: A in the strategic quadrant; B and D in quick wins; C and E in foundations first; F in park&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Value and feasibility matrix: A in the strategic quadrant; B and D in quick wins; C and E in foundations first; F in park" title="Value and feasibility matrix: A in the strategic quadrant; B and D in quick wins; C and E in foundations first; F in park" srcset="https://substackcdn.com/image/fetch/$s_!VtAX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png 424w, https://substackcdn.com/image/fetch/$s_!VtAX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png 848w, https://substackcdn.com/image/fetch/$s_!VtAX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png 1272w, https://substackcdn.com/image/fetch/$s_!VtAX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c05d785-3e69-47fa-b698-0fae561ba42b_1746x1023.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 5. Value and feasibility matrix for six requests (illustrative figures).</figcaption></figure></div><h3>6.2. Four quadrants, four responses</h3><p>Each quadrant in Figure 5 calls for a different response. <strong>Strategic</strong> (high value, feasible enough), here request A: invest seriously and accept a longer timeline. A's high value is realised only if the project measures the right thing, that is, targets the persuadable customers of the uplift model in Section 3.2 rather than sending offers to everyone with a high churn score. <strong>Quick wins</strong> (moderate value, easy), here B and D: do them early to build trust and to let the team learn. Bernard Marr advises a priority list of one to three big strategic opportunities plus one to three quick wins, because quick wins build confidence while larger projects are still under way (Marr, 2025). <strong>Foundations first</strong> (high value, not yet feasible), here C and E: do not drop them, but do not start now either; invest in the missing conditions, mostly data. <strong>Park</strong> (low value, hard) is F.</p><p>The matrix is not a decision machine. The scores are human judgements written down for transparency, so its greatest value is to expose those judgements to debate: if the sales director scores dynamic pricing 5 on financial impact and the finance director scores it 2, the reasons for the gap are what deserve discussion. Request F sitting close to the feasibility threshold is a reminder that the boundaries between quadrants are conventions; a small change in weights moves it to another quadrant.</p><h2>7. Business understanding within the CRISP-DM cycle</h2><p><a href="https://www.kdnuggets.com/2014/10/crisp-dm-top-methodology-analytics-data-mining-data-science-projects.html">CRISP-DM</a> (Cross-Industry Standard Process for Data Mining) is the most widely used process framework for structuring data-science projects (Chapman et al., 2000). It has six iterative phases that need not be followed rigidly in sequence, as Figure 6 shows.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!OpwB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png 424w, https://substackcdn.com/image/fetch/$s_!OpwB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png 848w, https://substackcdn.com/image/fetch/$s_!OpwB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png 1272w, https://substackcdn.com/image/fetch/$s_!OpwB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!OpwB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The CRISP-DM cycle in six steps: business understanding, data understanding, data preparation, modelling, evaluation, deployment&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The CRISP-DM cycle in six steps: business understanding, data understanding, data preparation, modelling, evaluation, deployment" title="The CRISP-DM cycle in six steps: business understanding, data understanding, data preparation, modelling, evaluation, deployment" srcset="https://substackcdn.com/image/fetch/$s_!OpwB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png 424w, https://substackcdn.com/image/fetch/$s_!OpwB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png 848w, https://substackcdn.com/image/fetch/$s_!OpwB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png 1272w, https://substackcdn.com/image/fetch/$s_!OpwB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71de7a3a-56ab-4f3b-bcc4-40c1af81e011_1679x1584.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 6. The six phases of the CRISP-DM cycle. Source: redrawn from Chapman et al. (2000).</figcaption></figure></div><p>Table 6 links each CRISP-DM phase to the Machine Learning Canvas blocks.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FrvSaF%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/930bcd68-78ef-47af-9182-fe69eae3c2b4_1220x1384.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/99525297-ff63-4584-98a5-fdcafb55a90e_1220x1504.png&quot;,&quot;height&quot;:677,&quot;title&quot;:&quot;Table 6. The six CRISP-DM phases and the corresponding ML Canvas blocks&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FrvSaF%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="677" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Table 6 shows that the canvas does not replace CRISP-DM but complements it: CRISP-DM describes <strong>the sequence of work</strong>, while the canvas records <strong>the design decisions</strong> on one page. Business understanding is the first phase of CRISP-DM and also the phase you return to after each deployment: every A/B test result feeds the questions for the next cycle.</p><h2>Conclusion</h2><p>Business understanding is not an unnecessary delay. It is the wisest investment you can make in a data project. By taking time to listen, ask questions and use structured tools such as the Machine Learning Canvas, you move from technician to strategic partner, someone who not only builds models but creates real value for the business.</p><p>The four questions sound simple, but each can transform a project: a "churn prediction model" can become a retention programme measured by experiment; a "recommendation system" may need only a few association rules; and a "95% accuracy" figure may find not a single customer.</p><p>Next time you receive a request, don't rush to open a Jupyter notebook. <strong>Open a conversation and a canvas.</strong> The success of the project, and sometimes saving the company from a costly mistake, depends on it.</p><h2>References</h2><p>Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C., &amp; Wirth, R. (2000). <em>CRISP-DM 1.0: Step-by-step data mining guide</em>. SPSS.</p><p>Dorard, L. (n.d.). <em>The Machine Learning Canvas</em>. <a href="https://www.ownml.co/machine-learning-canvas">https://www.ownml.co/machine-learning-canvas</a></p><p>Haughton, D., Haughton, J., &amp; Lo, V. S. Y. (2026). <em>Cause and effect business analytics and data science</em>. CRC Press.</p><p>Marr, B. (2025). <em>AI strategy: Unleash the power of artificial intelligence in your business</em>. Kogan Page. <a href="https://www.koganpage.com/business-strategy/ai-strategy-9781398619562">https://www.koganpage.com/business-strategy/ai-strategy-9781398619562</a></p><p>Moore, G. A. (2014). <em>Crossing the chasm</em> (3rd ed.). HarperBusiness. <a href="https://www.amazon.com/s?k=Crossing+the+chasm+Moore&amp;i=stripbooks">https://www.amazon.com/s?k=Crossing+the+chasm+Moore&amp;i=stripbooks</a></p><p>Radcliffe, N. J., &amp; Surry, P. D. (2011). <em>Real-world uplift modelling with significance-based uplift trees</em>. Stochastic Solutions White Paper. <a href="https://www.stochasticsolutions.com/pdf/sig-based-up-trees.pdf">https://www.stochasticsolutions.com/pdf/sig-based-up-trees.pdf</a></p>]]></content:encoded></item><item><title><![CDATA[Machine Learning Explained Through a Vietnamese Folk Tale]]></title><description><![CDATA[Supervised vs. unsupervised learning through the sparrows in Tam and Cam, with a confusion matrix and a Python grain-sorting experiment.]]></description><link>https://www.phuonglib.com/p/machine-learning-vietnamese-folk-tale</link><guid isPermaLink="false">https://www.phuonglib.com/p/machine-learning-vietnamese-folk-tale</guid><dc:creator><![CDATA[PHUONGLIB]]></dc:creator><pubDate>Tue, 17 Nov 2026 16:23:21 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TccI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!TccI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!TccI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!TccI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!TccI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Sparrows on a wicker tray in a sunlit courtyard sorting mixed grains into two neat heaps, one glowing orange in the sun.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Sparrows on a wicker tray in a sunlit courtyard sorting mixed grains into two neat heaps, one glowing orange in the sun." title="Sparrows on a wicker tray in a sunlit courtyard sorting mixed grains into two neat heaps, one glowing orange in the sun." srcset="https://substackcdn.com/image/fetch/$s_!TccI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!TccI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!TccI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!TccI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40bb4e6e-919d-4c1f-bf05-69fea36f1665_2560x1440.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><blockquote><p><strong>Key takeaways</strong></p><ul><li><p>In the Vietnamese tale of Tam and Cam, a fairy sends a flock of sparrows to separate paddy from rice. In data-science terms, the birds solve a <strong>classification</strong> problem.</p></li><li><p>If the fairy first shows the birds labelled examples, that is <strong>supervised learning</strong>: accurate and easy to evaluate, but it needs labelling work and is only as good as its training set.</p></li><li><p>If the birds group similar grains on their own and Tam names the piles, that is <strong>unsupervised learning</strong>, or clustering: no labels needed, but no guarantee the clusters match what you want.</p></li><li><p>A confusion matrix, precision and recall show how well the birds did; which error costs more is a business question, not a technical one.</p></li><li><p>A Python experiment shows that 20 labelled grains are enough for 99.7% accuracy on an easy problem, that scaling features cuts K-Means errors from 71 to 4, and that on a hard problem more labelled data stalls at about 73% unless the birds can see new features.</p></li><li><p>The story does not end when the basket is sorted. Next season the grains change, the model fails silently, and someone has to monitor it and retrain it.</p></li></ul></blockquote><div><hr></div><p>Once upon a time, in a fairy tale that has shaped generations of Vietnamese readers (Nguy&#7877;n &#272;&#7893;ng Chi, 2000), there lived a gentle girl named Tam who suffered at the hands of her stepmother and her stepsister Cam. Vietnam's own Cinderella story reaches its cruellest point when the village holds a festival: the stepmother mixes paddy (unhusked rice) with husked rice in one basket and tells Tam she may go only after sorting it, "paddy with paddy, rice with rice". The task seems impossible. As Tam sits weeping, B&#7909;t, the kindly old sage of Vietnamese folk tales, appears and, by magic, sends a flock of sparrows to help. In moments the birds have sorted every grain, and Tam can get ready for the festival. (In some versions the rice is mixed with beans rather than paddy; this article follows the popular version.)</p><p>The magic is wonderful. But what if we take off our fairy-tale glasses and look at the scene with the eyes of a twenty-first-century data scientist? The fairy's sparrows have, in essence, solved a <strong>classification</strong> problem: assigning each grain to one of two groups known in advance. And the way the fairy "directs" the flock maps neatly onto two foundational methods of machine learning: <strong>supervised learning</strong> and <strong>unsupervised learning</strong>.</p><p>The article "decodes" the magic in eight parts. Section 1 translates the story into the language of data. Section 2 retells the sorting as supervised learning, and Section 3 as unsupervised learning. Section 4 asks how accurately the birds sorted, using a confusion matrix. Section 5 tries both approaches in a few lines of Python, Section 6 discusses how many labelled grains are needed, Section 7 introduces other ways of learning beyond these two, and Section 8 follows the story into the next season, when the grains no longer look like last year's.</p><h2>1. Framing the data problem</h2><p>Before choosing a method, translate the story into data-science terms. Every data professional does this for every problem: name each component correctly. Table 1 shows the translation.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2Flq0Ac%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1d142b6e-7e4b-48cd-a5cb-8c57fc539c9a_1220x754.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bfa9de8d-2a6c-484d-9cda-9805ab9da0de_1220x874.png&quot;,&quot;height&quot;:391,&quot;title&quot;:&quot;Table 1. The tale of Tam and Cam in the language of data science&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:false}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2Flq0Ac%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="391" frameborder="0" scrolling="no"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>One could liken the fairy to a data engineer or a data scientist. Strictly, the role is closer to a data scientist: a data engineer builds the "pipelines" that get data to the right place, whereas here the fairy decides how to teach the birds and checks whether they got it right.</p><p>So which method did the fairy use to "train" the flock? There are two scenarios, and Figure 1 sets them side by side.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XjWc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png 424w, https://substackcdn.com/image/fetch/$s_!XjWc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png 848w, https://substackcdn.com/image/fetch/$s_!XjWc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png 1272w, https://substackcdn.com/image/fetch/$s_!XjWc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!XjWc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two processes: supervised learning goes from labelled data to training, prediction and evaluation; unsupervised learning goes from unlabelled data to clustering, interpretation and checking&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two processes: supervised learning goes from labelled data to training, prediction and evaluation; unsupervised learning goes from unlabelled data to clustering, interpretation and checking" title="Two processes: supervised learning goes from labelled data to training, prediction and evaluation; unsupervised learning goes from unlabelled data to clustering, interpretation and checking" srcset="https://substackcdn.com/image/fetch/$s_!XjWc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png 424w, https://substackcdn.com/image/fetch/$s_!XjWc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png 848w, https://substackcdn.com/image/fetch/$s_!XjWc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png 1272w, https://substackcdn.com/image/fetch/$s_!XjWc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4aa537b7-868f-4025-b477-c7090136a00f_1920x860.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 1. Two ways the fairy could teach the flock.</figcaption></figure></div><h2>2. Scenario one: supervised learning, with the fairy as a dedicated teacher</h2><p>In <strong>supervised learning</strong>, the model is "taught" with <strong>labelled data</strong>: every example comes with its answer, just as we study from a textbook with an answer key.</p><h3>2.1. The fairy's three steps</h3><p><strong>Step 1: prepare the training data.</strong> Instead of letting the birds rush in, the fairy takes a small handful from the basket and carefully "labels" each grain. He holds one up and tells the birds: <em>"This is paddy: longer, dark yellow husk, hard."</em> Then another: <em>"This is rice: shorter, white, translucent."</em> This set of clearly identified grains is the <strong>training set</strong>.</p><p><strong>Step 2: train the model.</strong> The birds "learn" from these examples, memorising the relationship between the <strong>features</strong> (size, shape, colour, hardness) and the <strong>labels</strong> (paddy, rice). The result is a rule, such as "long, dark grains are paddy", stored in the flock's "brain". That is the trained model.</p><p><strong>Step 3: deploy and classify.</strong> Having "graduated" from the fairy's course, the birds are released into the whole basket. For each grain, they look at its features, compare them with the learned rule and predict: <em>"This one is short and white&#8230; almost certainly rice."</em> Into the rice basket it goes, and so on until the whole basket is sorted.</p><h3>2.2. Strengths, weaknesses and a trap</h3><p>The strength of this approach is that the birds know exactly what they are looking for, so accuracy is usually high. And because there are answers, the fairy can <strong>evaluate</strong> the result objectively by counting how many grains were sorted correctly (Section 4).</p><p>The weakness is the up-front effort of building a labelled dataset. A few dozen grains are easy; millions of emails, or millions of medical images that need a doctor to read them, can make labelling very expensive.</p><p>There is a trap that few notice: <strong>a model is only as good as its training set</strong>. Suppose the basket contains a little brown rice, almost as dark as paddy, but the handful the fairy picked happened to contain only white rice. The birds have never seen brown rice and will easily drop it into the paddy basket. This is <strong>selection bias</strong>: the model learns well on its training data but fails on cases absent from it. In practice, many face-recognition systems make far more errors for groups that appear rarely in their training data.</p><p>But what if the fairy has no time to teach grain by grain? That is the second scenario.</p><h2>3. Scenario two: unsupervised learning, with the flock discovering on its own</h2><p>In <strong>unsupervised learning</strong>, the model must find structure and patterns in data <strong>without any labels</strong>, like being handed a box of multicoloured LEGO bricks and told to sort them into groups with no instructions.</p><h3>3.1. The flock's three steps</h3><p><strong>Step 1: receive raw data.</strong> The fairy simply appears, points at the basket and commands: <em>"Birds, put grains that look alike into groups!"</em> He never says what paddy or rice is. The birds have no information about the names of the grains.</p><p><strong>Step 2: cluster.</strong> The birds compare grains with one another by their features. One bird starts with a short, white grain and gathers others that are also short and white; another picks up a long, dark, hard grain and looks for similar ones. Gradually <strong>clusters</strong> form. This is <strong>clustering</strong>.</p><p><strong>Step 3: interpret the result.</strong> When the birds finish, Tam has several separate piles in front of her. The birds know only "pile one" and "pile two". It is Tam, a human, who looks and assigns meaning: <em>"Ah, this pile is all short, white grains, so it's rice. That one is long, dark grains, so it's paddy."</em></p><h3>3.2. Classification or clustering?</h3><p>Step 3 reveals an important distinction. The output of unsupervised learning is <strong>clustering</strong>, not classification. Classification assigns items to groups <strong>named in advance</strong>; clustering <strong>creates groups itself</strong>, and people name them afterwards. Table 2 compares the two scenarios.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FaiSig%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a6e02553-e6d0-4008-b548-9d1762504da2_1220x808.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f3f6aa5d-1901-4768-9d33-61da549483dc_1220x928.png&quot;,&quot;height&quot;:429,&quot;title&quot;:&quot;Table 2. Supervised and unsupervised learning in the tale of Tam and Cam&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FaiSig%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="429" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>3.3. Strengths and weaknesses</h3><p>The biggest strength is that no labelling is needed, and the model can find unexpected groups nobody thought of. The birds might create three clusters rather than two, and Tam discovers the third is empty, shrivelled grains, something the stepmother never mentioned. In business, this is how companies find customer segments they have never named.</p><p>The weakness is that nothing guarantees the clusters match the groups you need. Results depend heavily on which features are chosen and how "similarity" is measured. If the birds look only at colour, brown rice and paddy may end up in the same cluster. Section 5 shows how a small decision about measurement can change the result dramatically.</p><p>Whichever way the birds are taught, the final question remains: how well did they do?</p><h2>4. Evaluation: how many grains did the birds get right?</h2><p>In the story, the birds are perfect. In real life, no model is. Suppose the basket holds 1,000 grains, 500 paddy and 500 rice, and the birds' results are as in Table 3 (illustrative figures).</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FnSKyC%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1628849a-6da0-460c-812b-47c60bf1035b_1220x352.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3b7262b6-e42a-4330-aff0-74db54a81b19_1220x460.png&quot;,&quot;height&quot;:206,&quot;title&quot;:&quot;Table 3. The birds' confusion matrix (illustrative figures)&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FnSKyC%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="206" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Table 3 is a <strong>confusion matrix</strong>: each cell counts grains by the pair "true type &#8211; predicted type". Three common metrics follow:</p><ul><li><p><strong>Accuracy</strong> = (480 + 490) / 1,000 = <strong>97%</strong>: the share of grains sorted correctly.</p></li><li><p><strong>Precision</strong> of the rice basket: of the 510 grains the birds put in the rice basket, 490 really are rice, about <strong>96%</strong>. Precision answers: "How clean is the rice basket?"</p></li><li><p><strong>Recall</strong> of the rice basket: of the 500 real rice grains, the birds picked 490, or <strong>98%</strong>. Recall answers: "How much rice was missed?"</p></li></ul><p>For Tam, which error is worse: paddy in the rice basket (cooked rice with husks in it) or rice in the paddy basket (losing a little rice)? If Tam hates biting on paddy most, she needs very high precision for the rice basket, even at the cost of missing a little more rice. The question "which error is more expensive?" is a business question, not a technical one, and it decides which metric to prioritise. In spam filtering, letting a spam message through does less harm than blocking an important email; in cancer screening, missing a case does far more harm than a false alarm.</p><p>Enough theory. Let the computer play the birds.</p><h2>5. Trying it in Python</h2><p>The code below simulates 1,000 grains with two features: length (mm) and colour brightness (0 dark to 1 light). Paddy is assumed to be longer and darker than rice. The code then tries both approaches: supervised learning with logistic regression and unsupervised learning with the K-Means algorithm (scikit-learn developers, n.d.). Both techniques are covered in depth by James et al. (2021) and G&#233;ron (2022).</p><p>```python from collections import Counter import numpy as np from sklearn.cluster import KMeans from sklearn.linear_model import LogisticRegression from sklearn.preprocessing import StandardScaler</p><p>rng = np.random.default_rng(1)</p><p>paddy = np.column_stack([rng.normal(8.0, 0.6, 500), rng.normal(0.55, 0.08, 500)]) rice = np.column_stack([rng.normal(6.5, 0.5, 500), rng.normal(0.90, 0.05, 500)]) X = np.vstack([paddy, rice]) y = np.array(["paddy"] * 500 + ["rice"] * 500) X_scaled = StandardScaler().fit_transform(X)  # put both features on the same scale</p><p>sample = np.r_[0:10, 500:510] birds = LogisticRegression().fit(X_scaled[sample], y[sample]) print("Supervised - accuracy:", round((birds.predict(X_scaled) == y).mean(), 3))</p><p>for name, data in [("unscaled", X), ("scaled", X_scaled)]: clusters = KMeans(n_clusters=2, n_init=10, random_state=0).fit_predict(data) print("Unsupervised,", name) for k in (0, 1): print("  Cluster", k, ":", dict(Counter(y[clusters == k].tolist()))) ```</p><p>The output:</p><p>``<code> Supervised - accuracy: 0.997 Unsupervised, unscaled Cluster 0 : {'paddy': 38, 'rice': 467} Cluster 1 : {'paddy': 462, 'rice': 33} Unsupervised, scaled Cluster 0 : {'paddy': 4, 'rice': 500} Cluster 1 : {'paddy': 496} </code>``</p><p>Figure 2 plots the K-Means results in both cases.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!OICr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png 424w, https://substackcdn.com/image/fetch/$s_!OICr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png 848w, https://substackcdn.com/image/fetch/$s_!OICr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png 1272w, https://substackcdn.com/image/fetch/$s_!OICr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!OICr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two scatter plots of grain length and brightness: K-Means on unscaled data misassigns 71 grains, on scaled data only 4&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two scatter plots of grain length and brightness: K-Means on unscaled data misassigns 71 grains, on scaled data only 4" title="Two scatter plots of grain length and brightness: K-Means on unscaled data misassigns 71 grains, on scaled data only 4" srcset="https://substackcdn.com/image/fetch/$s_!OICr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png 424w, https://substackcdn.com/image/fetch/$s_!OICr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png 848w, https://substackcdn.com/image/fetch/$s_!OICr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png 1272w, https://substackcdn.com/image/fetch/$s_!OICr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c983186-18fe-41e3-9cc7-cc34681f2208_2018x959.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 2. K-Means before and after scaling the features. Simulated data.</figcaption></figure></div><p>Three lessons follow.</p><p><strong>First, with only 20 labelled grains, supervised learning classified 99.7% correctly.</strong> The two types of grain in the simulation differ clearly, so a few examples are enough for the birds to learn the boundary.</p><p><strong>Second, unsupervised learning still separated two groups without labels, but on unscaled data it misassigned 71 grains.</strong> The left half of Figure 2 shows why: K-Means measures "similarity" by distance, and length in millimetres (varying by a few mm) swamps brightness (varying by a few hundredths). The algorithm looks almost only at length and cuts the groups with a nearly vertical line, ignoring the very clear difference in colour.</p><p><strong>Third, after scaling both features to the same scale, K-Means misassigned only 4 grains.</strong> How you prepare the data matters as much as which algorithm you choose.</p><p>One final note: K-Means only returns "cluster 0" and "cluster 1". We know which cluster is rice because we compared it with the true labels, just as Tam had to look at the piles to name them. In practice, when there are no true labels to compare, interpreting clusters requires domain knowledge.</p><h2>6. How many sample grains does the fairy need?</h2><p>The example in Section 5 looks too easy: just 20 labelled grains and almost perfect accuracy. Is that always so? Figure 3 answers with a simulation: varying the number of labelled grains from 2 to 500, repeated 200 times with random handfuls, for two problems. The easy problem is paddy versus rice as above. The hard problem is two rice varieties with almost the same length and colour.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xi1j!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png 424w, https://substackcdn.com/image/fetch/$s_!xi1j!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png 848w, https://substackcdn.com/image/fetch/$s_!xi1j!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png 1272w, https://substackcdn.com/image/fetch/$s_!xi1j!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!xi1j!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Line chart: for paddy and rice, accuracy reaches 98% with just 2 labelled grains; for two similar rice varieties, accuracy rises from 60% to about 73% and plateaus&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Line chart: for paddy and rice, accuracy reaches 98% with just 2 labelled grains; for two similar rice varieties, accuracy rises from 60% to about 73% and plateaus" title="Line chart: for paddy and rice, accuracy reaches 98% with just 2 labelled grains; for two similar rice varieties, accuracy rises from 60% to about 73% and plateaus" srcset="https://substackcdn.com/image/fetch/$s_!xi1j!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png 424w, https://substackcdn.com/image/fetch/$s_!xi1j!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png 848w, https://substackcdn.com/image/fetch/$s_!xi1j!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png 1272w, https://substackcdn.com/image/fetch/$s_!xi1j!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfbe0379-bfe7-43ab-82c5-9587cf11ab14_1662x912.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 3. Number of labelled grains and correct classification rate for an easy and a hard problem. Simulated data.</figcaption></figure></div><p>Figure 3 shows two things. On the easy problem, just two sample grains (one of each) reach about 98%; more data barely helps. On the hard problem, accuracy rises from about 60% to about 70% with 20 grains, then <strong>plateaus around 73%</strong> even with hundreds more.</p><p>The second finding is the important lesson: when two types of grain really are alike in the features the birds can see, no amount of extra teaching overcomes that limit. To do better, the fairy must let the birds <strong>see new features</strong>: smell, stickiness when bitten, the shape of the grain's tip. Real projects are the same. When a model has plateaued, collecting more of the same data is usually less effective than finding new information about the objects. So "How much data do we need?" has no universal answer; it depends on how hard the problem is and how good the features are.</p><h2>7. Beyond these two ways of learning</h2><p>Supervised and unsupervised learning are the two foundational ways of learning, but not the only ones. Three others are used increasingly, and each can be retold with the fairy's story, as Table 4 shows.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FWv0Hq%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0bb23dcb-8bd6-4770-a40c-2249ba5ad0b9_1220x906.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/96b2369d-8096-4f9b-8046-bf6c8a79f35f_1220x976.png&quot;,&quot;height&quot;:423,&quot;title&quot;:&quot;Table 4. Three other ways of learning, retold through the tale&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FWv0Hq%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="423" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Self-supervised learning deserves a word more, because it underlies large language models such as ChatGPT, Claude and Gemini. Nobody labels billions of sentences; the model hides the next word and guesses it, and the text itself provides the answer. In form this is supervised learning (there is an answer to compare with); in origin it uses data nobody labelled. The boundaries between ways of learning are not as rigid as textbook definitions suggest.</p><p>The ways of learning differ mainly in one respect: how much the machine is told about the right answer. Choosing one therefore starts with asking what kind of data the business already has, not which algorithm is fashionable. With historical data and known answers, supervised learning applies; it creates most of the business value of machine learning today because it answers questions with measurable outcomes. With only unlabelled data, unsupervised learning can find groups, but an algorithm always returns clusters, even when they mean nothing commercially, and a person still decides which clusters deserve action. Reinforcement learning is harder to use than it looks. AlphaGo, the first program to beat a professional Go player on a full board, learned from human games and then from tens of millions of positions in games against itself (Silver et al., 2016). A business cannot try a million prices on real customers.</p><p>Table 5 summarises the main concepts for quick reference.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FvCggT%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d5e4abea-b22d-4b00-a635-1439ac4b22fc_1220x914.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/519bad28-3c02-4a74-bb61-15d663ba88a7_1220x984.png&quot;,&quot;height&quot;:457,&quot;title&quot;:&quot;Table 5. Key concepts&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FvCggT%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="457" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h2>8. Next season: the life cycle of a model</h2><p>The fairy tale ends when the birds finish sorting. A real machine-learning project does not: the day a model goes live is when most of the cost begins. This section carries the story into the next season.</p><h3>8.1. Six steps, one loop</h3><p>Figure 4 lays out the steps of a machine-learning project as a life cycle in which the last step loops back to the middle.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ex09!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png 424w, https://substackcdn.com/image/fetch/$s_!Ex09!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png 848w, https://substackcdn.com/image/fetch/$s_!Ex09!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png 1272w, https://substackcdn.com/image/fetch/$s_!Ex09!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ex09!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ff066337-95fd-427d-80b1-779869c7f5da_1672x660.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Six steps: problem, data, training, evaluation, deployment, monitoring; a red arrow loops from monitoring back to training when data drifts&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Six steps: problem, data, training, evaluation, deployment, monitoring; a red arrow loops from monitoring back to training when data drifts" title="Six steps: problem, data, training, evaluation, deployment, monitoring; a red arrow loops from monitoring back to training when data drifts" srcset="https://substackcdn.com/image/fetch/$s_!Ex09!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png 424w, https://substackcdn.com/image/fetch/$s_!Ex09!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png 848w, https://substackcdn.com/image/fetch/$s_!Ex09!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png 1272w, https://substackcdn.com/image/fetch/$s_!Ex09!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff066337-95fd-427d-80b1-779869c7f5da_1672x660.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 4. The life cycle of a machine-learning model. The red arrow is the retraining loop when data drifts.</figcaption></figure></div><p>The first five steps in Figure 4 match what this article has described: the fairy defines the problem (separating paddy from rice), picks a handful of sample grains, teaches the birds, counts their mistakes with a confusion matrix, and releases them on the real basket. The sixth step, monitoring, is the one a fairy tale does not need and every real system does. A group of Google engineers observed that in real machine-learning systems, the code that actually learns and predicts is a small fraction; the rest is data collection and verification, configuration, serving infrastructure and monitoring (Sculley et al., 2015). They called the hidden cost of shortcuts taken to ship fast <strong>technical debt</strong>: every shortcut today becomes maintenance cost tomorrow.</p><h3>8.2. When next season's grains are different</h3><p>Suppose that next year the stepmother switches to a new rice variety whose paddy is shorter and lighter, closer to husked rice. The birds keep sorting by what they learned last year, so they start leaving paddy in the rice bowl. Nobody raises an alarm: the birds sort as fast as ever, and only the rice bowl fills with more stray paddy. This is <strong>data drift</strong>: the world changes and the model does not. Models fail silently; they do not stop, they simply predict worse and worse.</p><p>For an online shop's fraud model, minimum monitoring has three parts. First, track the share of orders flagged each week, because a sudden change signals drift. Second, compare predictions with real labels when they arrive, for example when the bank reports chargebacks. Third, set in advance a drop in recall that triggers retraining. All three need a named owner, because a task that "everyone" looks after usually ends up looked after by no one.</p><h3>8.3. Questions before spending money</h3><p>Table 6 turns the concepts in this article into questions to ask a data team or vendor before approving a machine-learning project. A vague answer to any question marks something to clarify before spending money.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FB6CoY%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0886d26f-02e7-4083-b53e-66f5a2b07f0c_1220x1064.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8c265085-5c4b-4d22-9eb2-03619af565a5_1220x1134.png&quot;,&quot;height&quot;:469,&quot;title&quot;:&quot;Table 6. Six questions for a machine-learning project&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FB6CoY%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="469" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>The last question in Table 6 is the one most often skipped, because it never appears in a demo; it only shows up months after the model has gone live.</p><h2>Conclusion: what magic really happened?</h2><p>Given the perfect result and miraculous speed in the story, the fairy's magic most likely worked like <strong>supervised learning</strong>: the sparrows were a well-trained model that knew exactly what paddy and rice were. If the fairy wanted to challenge the flock a little more, he could have used <strong>unsupervised learning</strong>, letting the birds work out the basket's structure on their own and leaving Tam to name each pile.</p><p>But to retell the story accurately for today's data science, the real magic is not in the birds. It lies in what the fairy does before and after: choosing a representative handful of samples, deciding which features the birds should look at, measuring those features on the same scale, and checking whether the rice basket still has husks in it. Those are also the tasks that take up most of a data professional's time.</p><p>A seemingly simple folk tale holds a deep connection to the foundations of modern artificial intelligence. It shows that the idea of "teaching" another being to recognise and sort the world has long existed in human imagination. Next time you reread a fairy tale, you may find another technological idea hiding inside it.</p><h2>References</h2><p>G&#233;ron, A. (2022). <em>Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow</em> (3rd ed.). O'Reilly Media. <a href="https://www.oreilly.com/library/view/hands-on-machine-learning/9781098125967/">https://www.oreilly.com/library/view/hands-on-machine-learning/9781098125967/</a></p><p>James, G., Witten, D., Hastie, T., &amp; Tibshirani, R. (2021). <em>An introduction to statistical learning</em> (2nd ed.). Springer. <a href="https://www.statlearning.com/">https://www.statlearning.com/</a></p><p>Nguy&#7877;n &#272;&#7893;ng Chi. (2000). <em>Kho t&#224;ng truy&#7879;n c&#7893; t&#237;ch Vi&#7879;t Nam</em> [A treasury of Vietnamese folk tales]. Nh&#224; xu&#7845;t b&#7843;n Gi&#225;o d&#7909;c. <a href="https://www.amazon.com/s?k=Kho+t%C3%A0ng+truy%E1%BB%87n+c%E1%BB%95+t%C3%ADch+Vi%E1%BB%87t+Nam+Nguy%E1%BB%85n+%C4%90%E1%BB%95ng+Chi&amp;i=stripbooks">https://www.amazon.com/s?k=Kho+t%C3%A0ng+truy%E1%BB%87n+c%E1%BB%95+t%C3%ADch+Vi%E1%BB%87t+Nam+Nguy%E1%BB%85n+%C4%90%E1%BB%95ng+Chi&amp;i=stripbooks</a></p><p>scikit-learn developers. (n.d.). <em>Clustering: K-Means</em>. <a href="https://scikit-learn.org/stable/modules/clustering.html#k-means">https://scikit-learn.org/stable/modules/clustering.html#k-means</a></p><p>Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., &amp; Dennison, D. (2015). Hidden technical debt in machine learning systems. <em>Advances in Neural Information Processing Systems, 28</em>, 2503&#8211;2511. <a href="https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html">https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html</a></p><p>Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., &amp; Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. <em>Nature, 529</em>(7587), 484&#8211;489. <a href="https://doi.org/10.1038/nature16961">https://doi.org/10.1038/nature16961</a></p>]]></content:encoded></item><item><title><![CDATA[What Is Regression? Statistics and Machine Learning Compared]]></title><description><![CDATA[Regression in statistics vs. machine learning: linear and logistic models, explaining vs. predicting, R&#178;, MAE, RMSE and overfitting, with Python.]]></description><link>https://www.phuonglib.com/p/what-is-regression</link><guid isPermaLink="false">https://www.phuonglib.com/p/what-is-regression</guid><dc:creator><![CDATA[PHUONGLIB]]></dc:creator><pubDate>Tue, 17 Nov 2026 16:23:20 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0nui!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!0nui!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!0nui!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!0nui!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!0nui!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A young woman stretching an orange string through pebbles and flicking its end forward to land beside a pebble a wave just washed in.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A young woman stretching an orange string through pebbles and flicking its end forward to land beside a pebble a wave just washed in." title="A young woman stretching an orange string through pebbles and flicking its end forward to land beside a pebble a wave just washed in." srcset="https://substackcdn.com/image/fetch/$s_!0nui!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!0nui!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!0nui!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!0nui!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04a0169b-db8f-4a1a-87bf-820a444c55f8_2560x1440.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><blockquote><p><strong>Key takeaways</strong></p><ul><li><p>"Regression" comes from Francis Galton's observation that children's heights "regress" towards the average; regression to the mean is still a common thinking trap.</p></li><li><p>In statistics, regression is a family of techniques (linear, logistic, Poisson&#8230;). In machine learning, it is a type of task: predicting a continuous output, as opposed to classification.</p></li><li><p>Logistic regression is a regression technique in statistics but usually a classification algorithm in machine learning.</p></li><li><p>Explaining and predicting are different jobs. Explanatory models care about coefficients and need a proper research design before anyone talks about causation; predictive models care about error on data they have never seen.</p></li><li><p>A flexible model can fit training data perfectly and still predict worse than a simple line: always judge a predictive model on a holdout set.</p></li></ul></blockquote><div><hr></div><p>"What will revenue be next quarter?" "Which factors really drive purchase decisions?" "How likely is this customer to default?" Questions like these all lead to a familiar tool: <strong>regression</strong>.</p><p>But "regression" is used in two senses. When statisticians say "regression", they mean a family of <em>techniques</em>. When machine-learning engineers say it, they mean a <em>type of task</em>. The two uses share a mathematical foundation but serve different purposes: <strong>explaining</strong> a mechanism or <strong>predicting</strong> something unknown. Mixing up these purposes causes many mistakes in business analysis, from reading a coefficient as if it were a causal effect to showing off an "accurate" model only because it fits the data used to build it.</p><p>This article clarifies the difference. Section 1 tells the story of where the word "regression" came from. Section 2 separates the two uses of the term in statistics and machine learning. Section 3 introduces two basic regression techniques, linear and logistic regression, and Section 4 describes the regression task in machine learning. Section 5 is the heart of the article: how models built to explain differ from models built to predict, in the question they answer, how they are built and how they are evaluated, with a short Python example. Section 6 shows how to read the evaluation of a regression model correctly.</p><h2>1. Where does the word "regression" come from?</h2><p>In the late nineteenth century, Francis Galton measured the heights of parents and their children. He noticed that very tall parents tended to have tall children, but <em>less tall</em> than themselves; very short parents had short children, but <em>less short</em>. The next generation's height tended to "go back" towards the average. Galton (1886) called this <em>regression towards mediocrity</em>. The method he later used to draw a line describing the relationship between two variables took the name "regression", even though today it is used for all kinds of relationships, not only the "return to the average".</p><blockquote><p><strong>Regression to the mean</strong> is still a common thinking trap. A salesperson who has an unusually strong month usually has a weaker month next, even though nobody did anything wrong. If the company happened to "reprimand" that person in between, it is easy to conclude, wrongly, that reprimands don't work while praise "spoils" staff.</p></blockquote><h2>2. One word, two uses</h2><p>Today statisticians and machine-learning engineers both say "regression" but often do not mean the same thing. Table 1 puts the two uses side by side.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FIOSdR%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d2fc9f06-bb44-4ae2-9cab-001a571694ac_1220x660.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e2514af9-7d15-4461-a641-728dc1d798b5_1220x730.png&quot;,&quot;height&quot;:312,&quot;title&quot;:&quot;Table 1. \&quot;Regression\&quot; in statistics and in machine learning&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FIOSdR%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="312" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>The third row of Table 1 is the most confusing point: <strong>logistic regression</strong> has "regression" in its name but, in machine learning, is a <em>classification</em> algorithm. Conversely, the regression task in machine learning uses not only linear regression but also decision trees, random forests, gradient boosting, neural networks and more; many of these algorithms have versions for both classification and regression.</p><h2>3. Regression techniques in statistics</h2><p>Statistics offers many regression techniques, each suited to a type of dependent variable (James et al., 2021). This section introduces the two most basic and widely used in business analysis: linear regression for continuous variables and logistic regression for binary ones.</p><h3>3.1. Linear regression</h3><p>Linear regression assumes that the dependent variable <em>y</em> (a continuous variable such as revenue, price or temperature) changes along a straight line with the independent variables <em>x&#8321;, x&#8322;, &#8230;</em>:</p><p><em>y = &#946;&#8320; + &#946;&#8321;x&#8321; + &#946;&#8322;x&#8322; + &#8230; + &#949;</em></p><p>where <em>&#946;&#8320;</em> is the intercept, <em>&#946;&#8321;, &#946;&#8322;&#8230;</em> are the coefficients of each variable, and <em>&#949;</em> is the error the model cannot explain. The coefficients are usually estimated by <strong>ordinary least squares</strong> (OLS): choosing the line that minimises the sum of squared distances from the data points to the line. Figure 1 illustrates the idea with simulated data on advertising budget and sales: each point is a month, the red line is the estimated regression line, and the dashed segment is the error for one particular month.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!e0_8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png 424w, https://substackcdn.com/image/fetch/$s_!e0_8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png 848w, https://substackcdn.com/image/fetch/$s_!e0_8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png 1272w, https://substackcdn.com/image/fetch/$s_!e0_8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!e0_8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Scatter plot of advertising budget and sales with a regression line&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Scatter plot of advertising budget and sales with a regression line" title="Scatter plot of advertising budget and sales with a regression line" srcset="https://substackcdn.com/image/fetch/$s_!e0_8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png 424w, https://substackcdn.com/image/fetch/$s_!e0_8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png 848w, https://substackcdn.com/image/fetch/$s_!e0_8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png 1272w, https://substackcdn.com/image/fetch/$s_!e0_8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d70552b-328f-4344-8182-4ffe06e053e4_797x496.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 1. A linear regression line and the error of one observation (simulated data).</figcaption></figure></div><p><strong>Example (illustrative figures).</strong> A marketing director regresses monthly sales on the digital advertising budget and gets:</p><p><em>Sales (thousand USD) = 500 + 10 &#215; Advertising budget (thousand USD)</em></p><p>How to read it: with a zero budget, the model estimates sales of about USD 500,000; each additional USD 1,000 of advertising is <em>associated with</em> about USD 10,000 more in sales, holding other factors constant. Note the word "associated": if the company usually raises advertising exactly in the peak season, the coefficient of 10 may also be measuring the effect of seasonality (see Section 5).</p><h3>3.2. Logistic regression</h3><p>When the dependent variable is binary (yes/no, default/no default, 1/0), linear regression can produce "probabilities" below 0 or above 1. Logistic regression solves this by passing the linear combination through the <strong>sigmoid function</strong>, which keeps the result between 0 and 1:</p><p><em>P(y = 1) = 1 / (1 + e^&#8722;(&#946;&#8320; + &#946;&#8321;x&#8321; + &#8230;))</em></p><p><strong>Example (illustrative).</strong> A bank predicts the probability of default from income, current debt and credit history. The model does not answer "yes" or "no"; it gives a probability, say 0.85. The bank then chooses a <strong>threshold</strong> for its decision: a low threshold rejects many good customers, a high one accepts many bad ones. Choosing the threshold is a business decision, not the model's.</p><h2>4. The regression task in machine learning</h2><p>In machine learning, a <strong>regression task</strong> is a supervised learning task whose output is a continuous number: a house price, a delivery time, electricity consumption. When the data is large and the relationships are complex and non-linear, practitioners use algorithms such as regression trees, random forests, gradient boosting (XGBoost, LightGBM), support vector regression (SVR) or neural networks.</p><p><strong>Example.</strong> Ride-hailing and food-delivery apps such as Uber and Grab, Southeast Asia's leading super-app, have to estimate the <strong>estimated time of arrival (ETA)</strong>: the output is a number of minutes, so this is a regression task. The model uses many features, such as distance, time of day, traffic and weather. <strong>Surge pricing</strong> is more complex: it relies on forecasts of supply and demand by area (which may use regression models) plus rules and price-optimisation algorithms.</p><h2>5. Explain or predict?</h2><p>Shmueli (2010) showed that modelling <strong>to explain</strong> and modelling <strong>to predict</strong> are two different activities, even when they use the same equation. Breiman (2001) called this the "two cultures" of statistical modelling. Table 2 compares the two purposes on five aspects; the four subsections that follow cover each purpose, how to evaluate models and a worked example.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FfwLtN%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2dec2e5a-6044-42a8-b65d-f7a70489e73f_1220x766.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/47cdd249-7dbc-4db0-8dda-92de0cab41e0_1220x836.png&quot;,&quot;height&quot;:366,&quot;title&quot;:&quot;Table 2. Models to explain and models to predict&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FfwLtN%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="366" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>5.1. Regression to explain: correlation is not causation</h3><p>When regression is used to explain, researchers care about the coefficients and how reliable they are. But a regression equation by itself <strong>only measures how variables move together</strong>; it does not prove that one variable <em>causes</em> another.</p><p>The classic example: ice-cream sales are positively correlated with drownings. Selling ice cream does not cause drowning; both rise with <strong>summer temperatures</strong>, a confounder. Leaving out a confounder biases the regression coefficients; putting the wrong variable in can bias them too, as Simpson's paradox shows.</p><p>To conclude anything about causation, you need the right <strong>design</strong>: randomised controlled trials (RCTs), A/B tests, or causal inference methods such as difference-in-differences, instrumental variables and regression discontinuity. Regression is often still the calculating engine inside those designs, but it is the design that allows causal claims.</p><h3>5.2. Regression to predict: estimating the unknown</h3><p>"Prediction" is not just "forecasting the future". Prediction means using known data to estimate <strong>an unknown value</strong>, at any point in time:</p><ul><li><p><strong>The past:</strong> palaeontologists estimate the body mass of dinosaurs from the <strong>circumference</strong> of the femur and humerus, based on regression relationships measured in living animals (Campione &amp; Evans, 2012).</p></li><li><p><strong>The present:</strong> Zillow's Zestimate estimates the <em>current</em> price of a home that is not for sale. Central banks use <em>nowcasting</em> to estimate GDP growth for the current quarter from data released earlier (industrial output, retail sales, trade).</p></li><li><p><strong>The future:</strong> forecasting next month's sales or tomorrow's electricity demand. With stock prices, experience shows that models predicting short-term prices rarely beat a naive forecast consistently, a reminder that not everything can be predicted.</p></li></ul><p>In prediction tasks, the key question is how wrong the model is on new data. People using the Zestimate need a figure close to the real price; they care little about how many percentage points a red tiled roof adds.</p><h3>5.3. Evaluating models: all the data or a holdout?</h3><p>Because the two purposes differ, so does evaluation, especially on the question of which data to use to measure quality.</p><ul><li><p><strong>Explanatory models</strong> are usually estimated on the whole sample: a larger sample gives more precise coefficient estimates (smaller standard errors). Quality is judged by checking model assumptions, the meaning of the coefficients and their fit with theory.</p></li><li><p><strong>Predictive models</strong> must not be evaluated on the data used to train them. A sufficiently complex model can "memorise" even the noise, a problem called <strong>overfitting</strong>: the error on training data is tiny but the error on new data is large. The data is therefore split into a <strong>training set</strong> and a <strong>holdout (test) set</strong>, or <strong>cross-validation</strong> is used. With time-series data, the test set must come <em>after</em> the training set, never shuffled in at random.</p></li></ul><p>Table 3 summarises three common metrics for regression tasks.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F35QRG%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f41487de-7a66-40d5-a794-ac60e71d8b16_1220x586.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4499a514-8ab7-46be-a1b5-70b5dafeb9ec_1220x656.png&quot;,&quot;height&quot;:291,&quot;title&quot;:&quot;Table 3. Three metrics for evaluating regression models&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F35QRG%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="291" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>5.4. A short Python example</h3><p>The code below uses the same simulated data as Figure 1, splits it into a training set and a test set, and compares the errors of linear regression on the two sets.</p><p>```python import numpy as np from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score</p><p>rng = np.random.default_rng(42) budget = rng.uniform(10, 100, 200).reshape(-1, 1)            # thousand USD sales = 500 + 10 * budget.ravel() + rng.normal(0, 80, 200)   # illustrative</p><p>X_train, X_test, y_train, y_test = train_test_split( budget, sales, test_size=0.25, random_state=0)</p><p>model = LinearRegression().fit(X_train, y_train) print("Intercept:", round(model.intercept_, 1), "| Coefficient:", round(model.coef_[0], 2))</p><p>for name, X, y in [("Train", X_train, y_train), ("Test", X_test, y_test)]: pred = model.predict(X) print(name, "R2 =", round(r2_score(y, pred), 3), "| MAE =", round(mean_absolute_error(y, pred), 1), "| RMSE =", round(mean_squared_error(y, pred) ** 0.5, 1)) ```</p><p>The estimated coefficients are fairly close to 500 and 10 (about 469 and 10.55) because the data was generated from that very equation plus noise; the gap comes from the noise and the small sample. Table 4 compares the results of linear regression with a very flexible model, a decision tree with unlimited depth (<code>DecisionTreeRegressor()</code>).</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F74FHF%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/06025e3c-ed32-4253-8966-f549157b387d_1220x532.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dffe1f37-8c35-4844-923b-76ec8fb77a77_1220x652.png&quot;,&quot;height&quot;:288,&quot;title&quot;:&quot;Table 4. Errors of two models on the training and test sets (simulated data, thousand USD)&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F74FHF%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="288" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Table 4 shows that the decision tree "memorises" the training set completely (zero error) but predicts much worse on new data, while linear regression has similar errors on both sets. Figure 2 visualises the RMSE column of Table 4: the gap between the decision tree's two bars is the signature of overfitting.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!EcB8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png 424w, https://substackcdn.com/image/fetch/$s_!EcB8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png 848w, https://substackcdn.com/image/fetch/$s_!EcB8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png 1272w, https://substackcdn.com/image/fetch/$s_!EcB8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!EcB8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Bar chart of RMSE on the training and test sets for the two models&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Bar chart of RMSE on the training and test sets for the two models" title="Bar chart of RMSE on the training and test sets for the two models" srcset="https://substackcdn.com/image/fetch/$s_!EcB8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png 424w, https://substackcdn.com/image/fetch/$s_!EcB8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png 848w, https://substackcdn.com/image/fetch/$s_!EcB8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png 1272w, https://substackcdn.com/image/fetch/$s_!EcB8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe534f8f2-7e52-4daf-b1be-b49171f79af1_783x444.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 2. Overfitting: errors on the training and test sets for two models (simulated data).</figcaption></figure></div><h2>6. Reading a regression evaluation correctly</h2><p>Table 3 in Section 5.3 introduced the evaluation metrics. This section uses a small example to show how they differ, separates two kinds of interval that are often confused, and lists three biases that let a model score well on paper and still fail in use.</p><h3>6.1. Five days, four metrics</h3><p>A model forecasts the number of coffees a shop sells each day; on five days of the test set the actual counts are 120, 150, 90, 200 and 140 cups, and the forecasts are 110, 160, 100, 170 and 150 (illustrative figures). Figure 3 sets the two series side by side with each day's error.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KB9B!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png 424w, https://substackcdn.com/image/fetch/$s_!KB9B!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png 848w, https://substackcdn.com/image/fetch/$s_!KB9B!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png 1272w, https://substackcdn.com/image/fetch/$s_!KB9B!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!KB9B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Five pairs of bars, actual and forecast; errors of plus 10, minus 10, minus 10, plus 30 and minus 10 cups; day four is off the most&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Five pairs of bars, actual and forecast; errors of plus 10, minus 10, minus 10, plus 30 and minus 10 cups; day four is off the most" title="Five pairs of bars, actual and forecast; errors of plus 10, minus 10, minus 10, plus 30 and minus 10 cups; day four is off the most" srcset="https://substackcdn.com/image/fetch/$s_!KB9B!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png 424w, https://substackcdn.com/image/fetch/$s_!KB9B!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png 848w, https://substackcdn.com/image/fetch/$s_!KB9B!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png 1272w, https://substackcdn.com/image/fetch/$s_!KB9B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53214730-e8bd-4f44-9eb8-9d205f575282_1638x795.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 3. Five errors of a model forecasting daily coffee sales (illustrative figures).</figcaption></figure></div><p>From the five errors in Figure 3: <strong>MAE</strong> (mean absolute error) is (10 + 10 + 10 + 30 + 10) &#247; 5 = 14 cups. <strong>RMSE</strong> (root mean squared error) is the square root of (100 + 100 + 100 + 900 + 100) &#247; 5 = 260, about 16.1 cups; RMSE exceeds MAE because, once squared, the 30-cup error on day four weighs nine times as much as each 10-cup error. <strong>MAPE</strong> (mean absolute percentage error) is the average of 8.3%, 6.7%, 11.1%, 15.0% and 7.1%, about 9.7%. <strong>R&#178;</strong> is 1 minus the sum of squared errors (1,300) divided by the sum of squared deviations from the mean of 140 cups (6,600), about 0.80: compared with guessing 140 cups every day, the model removes about 80% of the squared error.</p><p>Choose the metric by the cost of mistakes. If running out of supplies on one day costs far more than a few days with slight surpluses, large errors are what matter and RMSE reflects that better than MAE. For a store manager, MAE ("off by 14 cups a day on average") is easiest to grasp. To compare accuracy across hundreds of items with very different volumes, MAPE gives a common scale, except for items with zero-sale days. It also pays to look at the signed mean error: here (10 &#8722; 10 &#8722; 10 + 30 &#8722; 10) &#247; 5 = 2 cups, so the model hardly leans to one side. And every number needs a benchmark: an MAE of 14 cups means something only next to a simple rule such as last week's sales on the same weekday. Two metrics can even pick different models: on a used-car price dataset, one method had the lower MAE but the higher RMSE, because it missed one car very badly (Camm et al., 2024).</p><h3>6.2. A confidence interval is not a prediction interval</h3><p>A regression model produces two kinds of interval, and reports often confuse them. Camm et al. (2024) use the example of a trucking company predicting that a 105-mile delivery with three stops takes 9.25 hours. The 95% <strong>confidence interval</strong> for the <em>mean</em> time of all such trips is only about 0.19 hours either side (9.06 to 9.44 hours). The 95% <strong>prediction interval</strong> for <em>one particular trip</em> is about 1.65 hours either side (7.61 to 10.90 hours), nearly nine times as wide, because besides the error in estimating the regression line there is the trip's own variability. A dispatcher scheduling this trip must look at the prediction interval. Both intervals widen as inputs move away from the range of the training data, so every prediction should come with an interval and a note of the range the data covered.</p><h3>6.3. Three biases that flatter evaluation scores</h3><p>Splitting the data correctly and choosing the right metric are still not enough. Table 5 lists three biases that can let a model score well on the test set and still fail in use.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FWAbY1%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d18435c7-ae84-4000-b0b2-1611bccc3c3c_1220x1098.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/41baec26-3a37-4a02-b61d-4510d4f7f848_1220x1218.png&quot;,&quot;height&quot;:472,&quot;title&quot;:&quot;Table 5. Three biases that make evaluation scores look better than reality&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FWAbY1%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="472" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Shortcut learning in Table 5 is well documented in machine-learning research (Geirhos et al., 2020). All three biases lead to one principle: an evaluation score is a conditional promise, valid only for the kind of data used to measure it.</p><h2>Conclusion</h2><p>The word "regression" was born from Galton's observation of heights "returning to the average" and became the name of both a family of statistical techniques and a type of machine-learning task. Understanding the two uses avoids common confusions, such as calling logistic regression a regression task, or assuming that every numerical prediction problem calls for linear regression.</p><p>More importantly, every time analysts run a regression model, they should ask themselves whether they are explaining or predicting. When explaining, the coefficients only show how variables move together, and causal claims need a proper research design. When predicting, the only trustworthy yardstick is the error on data the model has never seen. Two different questions demand two different ways of building and evaluating a model, even when the equation is identical.</p><h2>References</h2><p>Breiman, L. (2001). Statistical modeling: The two cultures. <em>Statistical Science, 16</em>(3), 199&#8211;231. <a href="https://doi.org/10.1214/ss/1009213726">https://doi.org/10.1214/ss/1009213726</a></p><p>Camm, J. D., Cochran, J. J., Fry, M. J., &amp; Ohlmann, J. W. (2024). <em>Business analytics</em> (5th ed.). Cengage Learning. <a href="https://www.cengage.com/c/business-analytics-5e-camm-cochran-fry-ohlmann/9780357902202/">https://www.cengage.com/c/business-analytics-5e-camm-cochran-fry-ohlmann/9780357902202/</a></p><p>Campione, N. E., &amp; Evans, D. C. (2012). A universal scaling relationship between body mass and proximal limb bone dimensions in quadrupedal terrestrial tetrapods. <em>BMC Biology, 10</em>, 60. <a href="https://doi.org/10.1186/1741-7007-10-60">https://doi.org/10.1186/1741-7007-10-60</a></p><p>DeGrave, A. J., Janizek, J. D., &amp; Lee, S.-I. (2021). AI for radiographic COVID-19 detection selects shortcuts over signal. <em>Nature Machine Intelligence, 3</em>(7), 610&#8211;619. <a href="https://doi.org/10.1038/s42256-021-00338-7">https://doi.org/10.1038/s42256-021-00338-7</a></p><p>Galton, F. (1886). Regression towards mediocrity in hereditary stature. <em>The Journal of the Anthropological Institute of Great Britain and Ireland, 15</em>, 246&#8211;263. <a href="https://doi.org/10.2307/2841583">https://doi.org/10.2307/2841583</a></p><p>Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., &amp; Wichmann, F. A. (2020). Shortcut learning in deep neural networks. <em>Nature Machine Intelligence, 2</em>(11), 665&#8211;673. <a href="https://doi.org/10.1038/s42256-020-00257-z">https://doi.org/10.1038/s42256-020-00257-z</a></p><p>James, G., Witten, D., Hastie, T., &amp; Tibshirani, R. (2021). <em>An introduction to statistical learning</em> (2nd ed.). Springer. <a href="https://www.statlearning.com/">https://www.statlearning.com/</a></p><p>Shmueli, G. (2010). To explain or to predict? <em>Statistical Science, 25</em>(3), 289&#8211;310. <a href="https://doi.org/10.1214/10-STS330">https://doi.org/10.1214/10-STS330</a></p>]]></content:encoded></item><item><title><![CDATA[The Secret to Long Life? Statistics Lessons from a Meme]]></title><description><![CDATA[A viral longevity meme exposes the classic traps of data analysis: bad data, confounders, survivorship bias, overfitting and the limits of R&#178;.]]></description><link>https://www.phuonglib.com/p/statistics-lessons-from-a-longevity-meme</link><guid isPermaLink="false">https://www.phuonglib.com/p/statistics-lessons-from-a-longevity-meme</guid><dc:creator><![CDATA[PHUONGLIB]]></dc:creator><pubDate>Tue, 17 Nov 2026 16:23:18 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Zz0b!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!Zz0b!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!Zz0b!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!Zz0b!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Zz0b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A rooster crowing on a wooden fence at dawn as a huge orange sun rises right behind it.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A rooster crowing on a wooden fence at dawn as a huge orange sun rises right behind it." title="A rooster crowing on a wooden fence at dawn as a huge orange sun rises right behind it." srcset="https://substackcdn.com/image/fetch/$s_!Zz0b!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!Zz0b!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!Zz0b!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!Zz0b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45eac696-44a9-4fa5-b235-776723b9d5f1_2560x1440.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><blockquote><p><strong>Key takeaways</strong></p><ul><li><p>A viral meme "shows" that every bad habit adds ten years of life. Checked against birth and death records, five of its six numbers are wrong, and two of its six people died in accidents.</p></li><li><p>Even with correct numbers, the meme confuses correlation with causation, ignores a large confounder (power and living conditions), allows for reverse causation and picks its sample to fit the story.</p></li><li><p>Fed to a regression model, the same six rows give R&#178; &#8776; 0.94 and a "prediction" that a healthy office worker will die at 30.5. A model can never be better than its data.</p></li><li><p>The Blue Zones studies offer a far more sensible list of nine habits, but they are still observational and deserve the same critical reading.</p></li><li><p>The same errors hide in business data. A bigger sample shrinks random error but never fixes a biased sampling frame, and four biases arise whenever we choose which data to look at.</p></li><li><p>Science moves from correlation to causation through experiments such as the PREDIMED trial: observation suggests, experiments test.</p></li></ul></blockquote><div><hr></div><p>Scrolling through social media one day, I came across a piece of "research" credited to a "Chinese Institute of Health". The image revealed a secret to long life that ran against every piece of health advice: each extra "vice" seemed to add another ten years.</p><p>Figure 1 reconstructs that meme in English. The original features six well-known figures from twentieth-century Chinese history, from the soldier-hero Lei Feng to the "Young Marshal" Zhang Xueliang.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XtOH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png 424w, https://substackcdn.com/image/fetch/$s_!XtOH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png 848w, https://substackcdn.com/image/fetch/$s_!XtOH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png 1272w, https://substackcdn.com/image/fetch/$s_!XtOH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!XtOH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;English reconstruction of the \&quot;secret to a long life\&quot; meme comparing the habits and lifespans of six Chinese historical figures&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="English reconstruction of the &quot;secret to a long life&quot; meme comparing the habits and lifespans of six Chinese historical figures" title="English reconstruction of the &quot;secret to a long life&quot; meme comparing the habits and lifespans of six Chinese historical figures" srcset="https://substackcdn.com/image/fetch/$s_!XtOH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png 424w, https://substackcdn.com/image/fetch/$s_!XtOH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png 848w, https://substackcdn.com/image/fetch/$s_!XtOH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png 1272w, https://substackcdn.com/image/fetch/$s_!XtOH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56226c41-e1a9-4297-9bee-332488430f3e_1920x1076.png 1456w" sizes="100vw"></picture><div></div></div></div><figcaption class="image-caption">Figure 1. The "secret to a long life" meme, reconstructed in English. Source: image circulated on social media, author unknown.</figcaption></figure></div><p>The "dataset" in the image is copied out in Table 1.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F1MrxQ%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5d54861d-a3f3-4433-b08b-bfa4df224150_1220x744.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9f88fb3c-e862-4480-941f-fcbd98b7b4e6_1220x814.png&quot;,&quot;height&quot;:370,&quot;title&quot;:&quot;Table 1. The \&quot;dataset\&quot; according to the meme&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:false}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F1MrxQ%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="370" frameborder="0" scrolling="no"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>The natural reaction: "So living a little badly is the real secret!" Not so fast. Before buying a pack of cigarettes, try on two different hats in turn: the hat of the <strong>statistical detective</strong>, who wants to know <em>why</em>, and the hat of the <strong>machine-learning prophet</strong>, who only wants to know <em>what will happen</em>.</p><p>The article has six parts. Section 1 follows the detective through four reasoning errors, starting with the most basic one: is the data even right? Section 2 hands the dataset to the machine-learning prophet and shows how useless a model that "fits very well" can be. Section 3 turns to a more serious question, what science says about living longer, through the Blue Zones research. Section 4 reads that research with the detective's eye, Section 5 shows how science moves from correlation to causation, and Section 6 carries the meme's lessons into business data.</p><h2>1. The statistical detective looks for the truth</h2><p>The detective's goal is a <strong>causal relationship</strong>: do smoking and drinking actually <em>make</em> people live longer? To answer, the detective has to rule out, one by one, the ways in which data can mislead.</p><h3>1.1. Error #0: is the data right?</h3><p>A detective's first job is to check the evidence. Comparing the birth and death years in standard biographies of these figures gives Table 2 and Figure 2.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FwR8HT%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7d9ebeff-37e9-4d6d-8039-3ec3bb80ea02_1220x840.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/889c18a8-2861-4114-80ea-550d26d70892_1220x960.png&quot;,&quot;height&quot;:413,&quot;title&quot;:&quot;Table 2. Lifespan according to the meme and according to birth and death years&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FwR8HT%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="413" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kwjr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png 424w, https://substackcdn.com/image/fetch/$s_!kwjr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png 848w, https://substackcdn.com/image/fetch/$s_!kwjr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png 1272w, https://substackcdn.com/image/fetch/$s_!kwjr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!kwjr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/daa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Chart comparing meme ages and real ages of the six figures; five of the six numbers are off, and two people died in accidents&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Chart comparing meme ages and real ages of the six figures; five of the six numbers are off, and two people died in accidents" title="Chart comparing meme ages and real ages of the six figures; five of the six numbers are off, and two people died in accidents" srcset="https://substackcdn.com/image/fetch/$s_!kwjr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png 424w, https://substackcdn.com/image/fetch/$s_!kwjr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png 848w, https://substackcdn.com/image/fetch/$s_!kwjr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png 1272w, https://substackcdn.com/image/fetch/$s_!kwjr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdaa60788-0f36-4dd9-be6a-dbc40818e566_1684x838.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 2. Age at death: according to the meme and according to birth and death years.</figcaption></figure></div><p>Five of the six numbers are off, one by as much as four years. To get its neat ten-year "law", the meme had to adjust the figures: the real ages are 63, 77, 82, 92 and 100, not 63, 73, 83, 93 and 103. And two of the six people died in accidents that had nothing to do with their habits.</p><p>If the input data is wrong, every analysis built on it is meaningless. In real projects this step is called <strong>data quality checking</strong>, and it often finds more problems than any model does. Tools such as Benford's law exist precisely to catch numbers that have been tampered with.</p><h3>1.2. Error #1: confusing correlation with causation</h3><p>Suppose, for the sake of argument, that the numbers were right. The next error is the most classic in statistics:</p><ul><li><p><strong>Correlation</strong>: two things tend to move together. In the meme, the number of "vices" is positively correlated with lifespan.</p></li><li><p><strong>Causation</strong>: one thing <em>causes</em> the other.</p></li></ul><p>A familiar example: in summer, ice-cream sales rise and so do drownings. Eating ice cream does not cause drowning; both rise because of <strong>hot weather</strong>. Hot weather here is a <strong>confounding variable</strong>: a third factor that affects both of the things being observed.</p><p>In the meme, the most obvious confounder is <strong>power and living conditions</strong>, as Figure 3 illustrates. These men had the best medical care of their time and could eat and rest far better than ordinary people. Those conditions helped them live longer and also allowed them to "indulge".</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5xGA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png 424w, https://substackcdn.com/image/fetch/$s_!5xGA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png 848w, https://substackcdn.com/image/fetch/$s_!5xGA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png 1272w, https://substackcdn.com/image/fetch/$s_!5xGA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!5xGA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Diagram: power and living conditions affect both indulgent habits and lifespan, creating a correlation; a reverse arrow from lifespan to habits shows reverse causation&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Diagram: power and living conditions affect both indulgent habits and lifespan, creating a correlation; a reverse arrow from lifespan to habits shows reverse causation" title="Diagram: power and living conditions affect both indulgent habits and lifespan, creating a correlation; a reverse arrow from lifespan to habits shows reverse causation" srcset="https://substackcdn.com/image/fetch/$s_!5xGA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png 424w, https://substackcdn.com/image/fetch/$s_!5xGA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png 848w, https://substackcdn.com/image/fetch/$s_!5xGA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png 1272w, https://substackcdn.com/image/fetch/$s_!5xGA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98e41ba-028a-43c6-b1bf-10cfaf134904_1610x845.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 3. Confounding and reverse causation in the meme.</figcaption></figure></div><p>Figure 3 also points to the opposite direction: you have to <strong>live long enough</strong> to have time to pick up more "habits" and a mistress. In other words, lifespan may be the cause of the number of "vices", not the other way round. This is called <strong>reverse causation</strong>.</p><h3>1.3. Error #2: selection bias and survivorship bias</h3><p>The "dataset" picks five long-lived people with bad habits and one person with a virtuous life who died young. It ignores the millions of smokers and drinkers who died at 50, and the millions of people with healthy habits who are blowing out the candles on their ninetieth birthday cake. This is <strong>selection bias</strong>: a sample chosen so that it yields the desired conclusion.</p><p>A special form is <strong>survivorship bias</strong>. During the Second World War, the Allies wanted to add armour to their bombers. They looked at the planes that came back and saw bullet holes concentrated on the wings, fuselage and tail. The statistician Abraham Wald argued the opposite: armour the places with <strong>no</strong> bullet holes, because planes hit there never made it back (Mangel &amp; Samaniego, 1984). Figure 4 illustrates the argument.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!F-fm!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp 424w, https://substackcdn.com/image/fetch/$s_!F-fm!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp 848w, https://substackcdn.com/image/fetch/$s_!F-fm!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp 1272w, https://substackcdn.com/image/fetch/$s_!F-fm!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!F-fm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Illustration of survivorship bias: bullet holes on returning planes; the areas without holes are the ones that need armour&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Illustration of survivorship bias: bullet holes on returning planes; the areas without holes are the ones that need armour" title="Illustration of survivorship bias: bullet holes on returning planes; the areas without holes are the ones that need armour" srcset="https://substackcdn.com/image/fetch/$s_!F-fm!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp 424w, https://substackcdn.com/image/fetch/$s_!F-fm!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp 848w, https://substackcdn.com/image/fetch/$s_!F-fm!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp 1272w, https://substackcdn.com/image/fetch/$s_!F-fm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c136722-fd66-4d1f-a7ee-06df6ee4d6c3_969x719.webp 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 4. Survivorship bias: the bullet holes we see and the ones we don't. Source: widely shared illustration, adapted from a figure by McGeddon on Wikimedia Commons (CC BY-SA 4.0).</figcaption></figure></div><p>Looking at Zhang Xueliang at 100 and concluding that drinking is the secret to long life is like looking at the holes in the wings and deciding to armour the wings. The people with "the full combo" who died early are the planes that never returned: they are not in the picture. Survivorship bias is everywhere, from "billionaires all dropped out of college" to "successful startups all started in a garage"; we only hear from those still standing to tell the story.</p><h3>1.4. Error #3: tiny sample and outliers</h3><p>The last two problems lie in the size of the dataset itself.</p><p><strong>Sample size.</strong> Serious studies of longevity follow thousands or hundreds of thousands of people over many years. Here the sample size is six. With six people, almost any "law" can appear by chance, and swapping a single person can reverse it.</p><p><strong>Outliers.</strong> Lei Feng and Lin Biao both died in accidents. Including them in a comparison of lifestyles is like concluding "if you want to live long, don't walk under poles and don't fly".</p><p><strong>The detective's verdict:</strong> wrong data, a hand-picked sample, a sample size of six, two outliers, a large confounder and the possibility of reverse causation. The meme proves nothing.</p><p>But the detective is only one of two people reading the data. The second one does not care about causes.</p><h2>2. The machine-learning prophet only cares about prediction</h2><p>Now put on the hat of a machine-learning engineer. This person asks a different question: <strong>I don't need to know why; I just need to predict correctly.</strong> Given enough data, this is a <strong>regression</strong> problem: the target variable is <code>lifespan</code>, and the features are <code>smokes</code>, <code>drinks</code>, <code>plays_cards</code> and <code>has_mistress</code>.</p><h3>2.1. Encoding the data</h3><p>The first step is to turn descriptions into numbers: each habit is 1 if present and 0 if not, as in Table 3. To see what the model learns from the data it is given, the table uses the meme's own numbers.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FqXrTH%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f77f933f-1461-4fbb-a8cc-ec344ee708b7_1220x750.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/772bb586-5273-4c93-b2a7-ff3143831319_1220x820.png&quot;,&quot;height&quot;:387,&quot;title&quot;:&quot;Table 3. The meme data after encoding&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FqXrTH%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="387" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>2.2. Trying it in Python</h3><p>A linear regression model learns a formula of the form:</p><p><code>lifespan = w0 + w1&#183;smokes + w2&#183;drinks + w3&#183;plays_cards + w4&#183;has_mistress</code></p><p>The code below estimates that formula and then uses it to predict the lifespan of a person with none of the habits.</p><p>```python import numpy as np from sklearn.linear_model import LinearRegression</p><p>X = np.array([[1, 0, 0, 0], [0, 1, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1], [0, 0, 0, 0]]) y = np.array([63, 73, 83, 93, 103, 23])</p><p>m = LinearRegression().fit(X, y) print("R2 on training data:", round(m.score(X, y), 3)) print("Coefficients:", np.round(m.coef_, 1), "| Intercept:", round(m.intercept_, 1))</p><p>office_worker = np.array([[0, 0, 0, 0]])  # an office worker who neither smokes nor drinks print("Predicted lifespan:", round(m.predict(office_worker)[0], 1)) ```</p><p>The output:</p><p>``<code> R2 on training data: 0.944 Coefficients: [25.  35.   2.5 10. ] | Intercept: 30.5 Predicted lifespan: 30.5 </code>``</p><p>Figure 5 plots the coefficients the model "learned".</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!25ul!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png 424w, https://substackcdn.com/image/fetch/$s_!25ul!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png 848w, https://substackcdn.com/image/fetch/$s_!25ul!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png 1272w, https://substackcdn.com/image/fetch/$s_!25ul!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!25ul!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Bar chart of regression coefficients: smoking adds 25 years, drinking 35 years, playing cards 2.5 years, a mistress 10 years, and the intercept is 30.5 years&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Bar chart of regression coefficients: smoking adds 25 years, drinking 35 years, playing cards 2.5 years, a mistress 10 years, and the intercept is 30.5 years" title="Bar chart of regression coefficients: smoking adds 25 years, drinking 35 years, playing cards 2.5 years, a mistress 10 years, and the intercept is 30.5 years" srcset="https://substackcdn.com/image/fetch/$s_!25ul!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png 424w, https://substackcdn.com/image/fetch/$s_!25ul!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png 848w, https://substackcdn.com/image/fetch/$s_!25ul!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png 1272w, https://substackcdn.com/image/fetch/$s_!25ul!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F662f6907-ab7f-49fb-926d-2017c7a6a6a8_1662x997.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 5. The coefficients a regression model learns from the six rows of meme data.</figcaption></figure></div><p>R&#178; is about 0.94: the model explains 94% of the variation in lifespan in the data, a number that looks very "nice". It "concludes" that smoking adds 25 years of life, drinking adds 35 and a mistress adds 10. And it predicts that a 30-year-old office worker who neither smokes nor drinks will live to&#8230; 30.5, that is, only a few more months.</p><h3>2.3. Proxy variables: right for the wrong reason</h3><p>The feature <code>has_mistress</code> appears only for the longest-lived person, so it becomes a "strong" signal. But it is a <strong>proxy variable</strong>: not a cause, but a stand-in for something hidden behind it, here power and wealth, the very confounder in Figure 3.</p><p>Proxies are common in business. The number of baby strollers sold at a store predicts nappy sales well, even though strollers do not create the need for nappies; both reflect how many families with young children live nearby. Proxies are useful for prediction but dangerous when read as causes, and they can become a back door to discrimination when they stand in for a sensitive attribute. A credit-scoring model that does not use "ethnicity" but does use "postcode" can still discriminate by ethnicity if where people live reflects their ethnicity.</p><h3>2.4. When the prophet fails</h3><p>The model in Section 2.2 makes two classic machine-learning mistakes.</p><p><strong>Overfitting.</strong> With six rows of data and five coefficients to estimate, the model all but "memorises" the data instead of learning a general pattern, like a student who memorises the answers to six worked examples but cannot solve a new problem. A high R&#178; on training data is therefore a <strong>warning sign</strong>, not a reason to celebrate.</p><p><strong>Failure to generalise.</strong> A model learned from six twentieth-century Chinese politicians cannot be applied to anyone else. Fit on training data says nothing about new data; that requires a separate test set the model has never seen.</p><p><strong>The prophet's verdict:</strong> "I have a model that fits very well&#8230; for exactly these six people. Don't use it on anyone else."</p><p>Table 4 puts the two hats side by side.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FjdHmX%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ffa0fc78-6e34-4272-b508-8e91b90ab2c0_1220x532.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d047906d-f129-43fb-a8c0-c6392b900cc7_1220x652.png&quot;,&quot;height&quot;:295,&quot;title&quot;:&quot;Table 4. The statistical detective and the machine-learning prophet&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FjdHmX%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="295" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Both hats conclude that the meme is worthless, for different reasons. So the original question remains: what does science say about the secret to a long life?</p><h2>3. The Blue Zones and their nine habits</h2><p>The journalist Dan Buettner, working with National Geographic and demographers, studied five regions with unusually many people living past 90 and 100: Okinawa (Japan), Sardinia (Italy), Nicoya (Costa Rica), Ikaria (Greece) and the Seventh-day Adventist community in Loma Linda (United States). He called them the <strong>Blue Zones</strong> and summarised nine shared habits, the <strong>Power 9</strong> (Buettner, 2012). Table 5 summarises the nine habits in four groups.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F2xD7g%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a5c16680-e754-4e24-97ed-8f7567eb287e_1220x1526.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/74c93e1c-f386-4c78-a574-1ad93a3a32e9_1220x1596.png&quot;,&quot;height&quot;:748,&quot;title&quot;:&quot;Table 5. The nine Blue Zones habits according to Buettner&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2F2xD7g%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="748" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>This list sounds far more sensible than the meme. But after Section 1 you have the tools to ask: how was it built, and how certain is it?</p><h2>4. Reading the Blue Zones with a detective's eye</h2><p><strong>These are observational studies, not experiments.</strong> People in the Blue Zones differ from people elsewhere in many ways: genes, environment, history, income, isolation. The effect of each habit cannot be isolated. The Power 9 is therefore a set of <strong>hypotheses</strong> worth testing, not a proven formula. This is Error #1 from Section 1.2, on a larger and subtler scale.</p><p><strong>Are the ages right?</strong> The demographer Saul Newman (2019) showed that many of the world's longevity records cluster in places with incomplete birth records, high poverty and incentives to misreport age, for example to claim a pension; many civil records in Okinawa were destroyed in the Second World War. This is Error #0 on a larger scale. Newman's findings are contested among demographers, but they are a reminder that data on the very old needs careful checking, just like the numbers in the meme.</p><p><strong>On alcohol.</strong> The World Health Organization (2023) states that no level of alcohol consumption is safe for health. Some Sardinians drink wine every day and still live long, but that may be thanks to other factors, such as movement, community or genes, rather than the wine. That is a confounder again, and possibly survivorship bias too: drinkers who did not live long are not among the elderly people interviewed.</p><p><strong>The Blue Zones are also a brand.</strong> The "Blue Zones" name is used for books, community programmes and commercial products. There is nothing wrong with that, but it is worth knowing when you read, because those who use a brand have a motive to reach appealing conclusions.</p><h2>5. When a hypothesis is put to the test</h2><p>Not every habit in the Power 9 remains a hypothesis. Some have been tested in experiments, the only tool that gets past all four errors in Section 1.</p><p>A prominent example is the <strong>PREDIMED</strong> trial in Spain. Researchers randomly assigned nearly 7,500 people at high cardiovascular risk to one of three diets. The groups following a Mediterranean diet supplemented with extra-virgin olive oil or nuts had about 30% fewer major cardiovascular events than the control group, which was advised to eat a low-fat diet (Estruch et al., 2018). Because of random assignment, the groups were alike on average in power, income, genes and every other confounder; the only systematic difference was the diet.</p><p>This is how science moves from correlation to causation: <strong>observation suggests, experiments test.</strong> The meme stopped at "observation" with a wrong and hand-picked sample. The Blue Zones went further with more careful observation, but it is still observation. PREDIMED is the testing step, and only for one specific habit.</p><h2>6. From a meme to business data</h2><p>The meme misleads because it shows a few long-lived men and hides everyone who died young. Business data make exactly the same mistakes; they are just harder to see in a spreadsheet with thousands of rows. This section gathers the lessons into four biases that arise when choosing data.</p><h3>6.1. A bigger sample does not fix bias</h3><p><strong>Bias</strong> is a systematic difference, repeated in the same direction, between what the data show and what is true. "Systematic" is what separates it from random error: random error goes up and down and shrinks as the sample grows, while bias does not shrink. Figure 6 illustrates with a hypothetical supermarket chain where 70% of customers are satisfied. If the chain surveys through an in-app notification, the sampling frame includes only customers who installed the app, a group that is more satisfied (80%), while older customers paying cash never have a chance to be sampled, however perfectly random the draw within that list.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SH_k!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png 424w, https://substackcdn.com/image/fetch/$s_!SH_k!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png 848w, https://substackcdn.com/image/fetch/$s_!SH_k!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png 1272w, https://substackcdn.com/image/fetch/$s_!SH_k!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!SH_k!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two bands by sample size from 50 to 5,000: samples of all customers converge on 70%, samples of app users converge on 80%; both bands narrow, but the second never reaches 70%&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two bands by sample size from 50 to 5,000: samples of all customers converge on 70%, samples of app users converge on 80%; both bands narrow, but the second never reaches 70%" title="Two bands by sample size from 50 to 5,000: samples of all customers converge on 70%, samples of app users converge on 80%; both bands narrow, but the second never reaches 70%" srcset="https://substackcdn.com/image/fetch/$s_!SH_k!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png 424w, https://substackcdn.com/image/fetch/$s_!SH_k!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png 848w, https://substackcdn.com/image/fetch/$s_!SH_k!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png 1272w, https://substackcdn.com/image/fetch/$s_!SH_k!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5e158212-b710-436d-b13e-ad17b658f88e_1620x869.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Figure 6. A larger sample narrows random error but cannot fix a biased sampling frame (illustrative figures).</figcaption></figure></div><p>Figure 6 shows that a large biased sample only delivers a wrong conclusion with more confidence. The 1936 <em>Literary Digest</em> poll is the classic lesson: the magazine mailed about ten million ballots, received about 2.4 million, and predicted a win for Landon, while Roosevelt won about 61% of the popular vote. Later analysis found that the sampling frame was skewed, but most of the error came from Landon supporters being keener to send their ballots back (Squire, 1988). More than two million responses could not rescue a biased sample.</p><h3>6.2. Four biases in choosing data</h3><p>Table 6 sets the four biases side by side, with warning signs and remedies.</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FfajWd%2F&amp;key=e27c740634285c9ddc20db64f73358dd&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/853a393f-79d9-4f00-a7cd-9ca503290ea6_1220x1044.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/acbb53a8-7d01-4014-91d8-477fe01fb1fb_1220x1114.png&quot;,&quot;height&quot;:460,&quot;title&quot;:&quot;Table 6. Four biases that arise when choosing data&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://iframely.net/api/iframe?app=1&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FfajWd%2F&amp;key=e27c740634285c9ddc20db64f73358dd" width="730" height="460" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Two rows of Table 6 deserve a closer look. On nonresponse bias, a common misconception is that a low response rate ruins the results. A synthesis of nearly sixty studies found that response rates on their own predict bias poorly (Groves &amp; Peytcheva, 2008); bias appears only when the decision to respond is related to the quantity being measured, as when the angriest customers are the ones who skip a satisfaction survey. On survivorship bias, Abraham Wald's aircraft from Section 1.3 are everywhere in business: an analysis of "what three-year loyal customers have in common" sees only the customers who stayed; backtesting an investment strategy on currently listed shares ignores companies delisted after losses. The check for any data file is the question the statistical detective asked of the meme: what process put this row in the file, and who was removed before I saw it?</p><h2>Conclusion: what one meme can teach</h2><p>One viral image contains almost every trap in data analysis: tampered numbers, a hand-picked sample, a sample of six, outliers, a confounder, reverse causation and survivorship bias. Hand it to a machine-learning model and you get a lovely R&#178; along with an absurd prediction, because a model can never be better than the data it is given.</p><p>The biggest lesson: <strong>tools are only tools; critical thinking and an understanding of where data comes from decide the result.</strong> The statistical detective and the machine-learning prophet ask different questions, but both must start with the same one: where does this data come from, and is it right?</p><p>Next time you see a "secret" built on a few hand-picked examples, whether it is the secret to living long, getting rich or building a startup, remember this story. Then eat some more vegetables, take a walk and call someone you love: those things, at least, need no meme to prove they are worth doing.</p><h2>References</h2><p>Buettner, D. (2012). <em>The blue zones: 9 lessons for living longer from the people who've lived the longest</em> (2nd ed.). National Geographic. <a href="https://www.amazon.com/s?k=The+blue+zones+Buettner&amp;i=stripbooks">https://www.amazon.com/s?k=The+blue+zones+Buettner&amp;i=stripbooks</a></p><p>Estruch, R., Ros, E., Salas-Salvad&#243;, J., Covas, M.-I., Corella, D., Ar&#243;s, F., &#8230; Mart&#237;nez-Gonz&#225;lez, M. A. (2018). Primary prevention of cardiovascular disease with a Mediterranean diet supplemented with extra-virgin olive oil or nuts. <em>New England Journal of Medicine, 378</em>(25), e34. <a href="https://doi.org/10.1056/NEJMoa1800389">https://doi.org/10.1056/NEJMoa1800389</a></p><p>Groves, R. M., &amp; Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. <em>Public Opinion Quarterly, 72</em>(2), 167&#8211;189. <a href="https://doi.org/10.1093/poq/nfn011">https://doi.org/10.1093/poq/nfn011</a></p><p>Mangel, M., &amp; Samaniego, F. J. (1984). Abraham Wald's work on aircraft survivability. <em>Journal of the American Statistical Association, 79</em>(386), 259&#8211;267. <a href="https://doi.org/10.1080/01621459.1984.10478038">https://doi.org/10.1080/01621459.1984.10478038</a></p><p>Newman, S. J. (2019). <em>Supercentenarians and the oldest-old are concentrated into regions with no birth certificates and short lifespans</em>. bioRxiv. <a href="https://doi.org/10.1101/704080">https://doi.org/10.1101/704080</a></p><p>Squire, P. (1988). Why the 1936 Literary Digest poll failed. <em>Public Opinion Quarterly, 52</em>(1), 125&#8211;133. <a href="https://doi.org/10.1086/269085">https://doi.org/10.1086/269085</a></p><p>World Health Organization. (2023, January 4). <em>No level of alcohol consumption is safe for our health</em>. <a href="https://www.who.int/europe/news/item/04-01-2023-no-level-of-alcohol-consumption-is-safe-for-our-health">https://www.who.int/europe/news/item/04-01-2023-no-level-of-alcohol-consumption-is-safe-for-our-health</a></p>]]></content:encoded></item><item><title><![CDATA[Simpson's Paradox: When the Average Misleads You]]></title><description><![CDATA[Why the same data can support opposite conclusions when aggregated and when split by group, and how to avoid being fooled by it]]></description><link>https://www.phuonglib.com/p/simpsons-paradox-when-the-average-misleads-you</link><guid isPermaLink="false">https://www.phuonglib.com/p/simpsons-paradox-when-the-average-misleads-you</guid><dc:creator><![CDATA[PHUONGLIB]]></dc:creator><pubDate>Tue, 17 Nov 2026 13:07:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/be18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6Tg-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6Tg-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!6Tg-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!6Tg-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!6Tg-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!6Tg-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp" width="2560" height="1440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/be18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1440,&quot;width&quot;:2560,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Simpson's Paradox: When the Average Misleads You&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Simpson's Paradox: When the Average Misleads You" title="Simpson's Paradox: When the Average Misleads You" srcset="https://substackcdn.com/image/fetch/$s_!6Tg-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp 424w, https://substackcdn.com/image/fetch/$s_!6Tg-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp 848w, https://substackcdn.com/image/fetch/$s_!6Tg-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp 1272w, https://substackcdn.com/image/fetch/$s_!6Tg-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe18a176-984a-4992-8ddb-1166dc7b7f7a_2560x1440.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><blockquote><p><strong>Key takeaways</strong></p><ul><li><p>Simpson's paradox occurs when a trend holds in every group but weakens or reverses once the groups are combined. Both sets of figures are arithmetically correct.</p></li><li><p>It needs three conditions at once: a lurking variable that affects the outcome, a different group composition between the options, and enough imbalance to tip the weighted average.</p></li><li><p>The data alone cannot tell you whether to trust the aggregated or the stratified figures. The causal role of the grouping variable (confounder, mediator or collider) decides.</p></li><li><p>A regression coefficient has no causal meaning until the causal structure behind it is stated.</p></li><li><p>Prevent it by design (randomization, stratified randomization, SRM checks), detect it in analysis (compare group composition first), and choose the figure that matches the question.</p></li></ul></blockquote><div><hr></div><p>Imagine you run a hospital. The report on your desk compares two treatments for kidney stones: treatment A succeeds <strong>78%</strong> of the time, treatment B <strong>83%</strong>. The choice looks clear: pick B.</p><p>Then a doctor on the board asks to split the figures by stone size. The result silences the room: <strong>for small stones, A beats B; for large stones, A also beats B</strong>. How can A win in every group and still lose when the groups are combined?</p><p>This is <strong>Simpson's paradox</strong>. It has reversed conclusions in medicine, education, sports, business and even lawsuits. This article answers four questions in turn: what Simpson's paradox is and why it happens; where it shows up in practice; whether to trust the aggregated or the stratified figures when it appears; and how to detect, handle and prevent it systematically. Section 1 defines the paradox and reviews its history. Section 2 dissects the kidney stone example, and Section 3 presents five real-world cases. Section 4 tackles the key question: aggregate or stratify, with a business example read through a causal diagram. The next three sections cover prevention through study design, detection during analysis, and correct handling once the paradox is found. Section 8 summarizes where the paradox tends to appear in each field, and Section 9 answers some common questions.</p><h2>1. What is Simpson's paradox?</h2><p><strong>Simpson's paradox</strong> occurs when a trend appears consistently within each group of data but <strong>weakens, disappears or reverses</strong> when the groups are combined (Simpson, 1951; Blyth, 1972).</p><p>It is not a logical contradiction. Both the aggregated and the stratified figures are arithmetically correct. What gets fooled is <strong>intuition</strong>: we assume that if A beats B in every group, A must beat B overall. That intuition fails in one specific situation, when the options being compared have <strong>different group compositions</strong>. The next section shows exactly how.</p><h3>1.1. A brief history</h3><p>The paradox bears Simpson's name, but Simpson was not the first to notice it. The four milestones below show that it was recognized as early as the end of the 19th century.</p><ul><li><p><strong>1899:</strong> Pearson and colleagues noted that mixing heterogeneous groups can produce spurious correlations (Pearson et al., 1899).</p></li><li><p><strong>1903:</strong> Yule analysed similar cases in his work on the association of attributes (Yule, 1903). For this reason the phenomenon is sometimes called the <em>Yule&#8211;Simpson effect</em>.</p></li><li><p><strong>1951:</strong> Edward H. Simpson published a detailed analysis of multi-way contingency tables (Simpson, 1951).</p></li><li><p><strong>1972:</strong> Colin Blyth called it "Simpson's paradox", and the name has stuck ever since (Blyth, 1972).</p></li></ul><h2>2. Anatomy of an example: treating kidney stones</h2><p>The opening scenario is based on real data. Charig et al. (1986) compared treatments for kidney stones; Julious and Mullee (1994) later used the comparison between open surgery (A) and percutaneous nephrolithotomy (B) to illustrate Simpson's paradox. This was an <strong>observational</strong> study: doctors chose the treatment for each patient, with no random allocation (Table 1).</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/qeW6N/2/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6aa0dbbd-8cc3-400d-a7bc-e388fb4544f8_1220x394.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8c6b6146-77a0-4410-8372-8b629c6149e0_1220x464.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 1. Kidney stone treatment success rates by stone type&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/qeW6N/2/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>2.1. Step 1: look at the case mix</h3><p>The key lies in which patients each treatment received (Table 2):</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/3N76N/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b7ecfdb5-3c0d-4eae-8cc8-e4a0335774cc_1220x384.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8fb78882-0fa5-4103-9149-53784e2155af_1220x454.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 2. Case mix of the two treatments&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/3N76N/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Treatment A was used mostly on <strong>hard cases</strong>, B mostly on <strong>easy cases</strong>. Large stones are harder to treat whichever method is used, so A's overall rate is pulled down. B handled mostly easy cases, so its overall figure looks good, even though it is actually worse than A for both stone sizes.</p><p><strong>Stone size is the lurking variable</strong>: it influences both which treatment the doctor chooses and the chance of success.</p><p>Figure 1 puts the rates from Table 1 side by side: A is higher than B for both stone sizes but lower in the overall column.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!d9Ah!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!d9Ah!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png 424w, https://substackcdn.com/image/fetch/$s_!d9Ah!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png 848w, https://substackcdn.com/image/fetch/$s_!d9Ah!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png 1272w, https://substackcdn.com/image/fetch/$s_!d9Ah!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!d9Ah!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png" width="783" height="426" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:426,&quot;width&quot;:783,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:29223,&quot;alt&quot;:&quot;Bar chart of the success rates of the two treatments by stone type and overall&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Bar chart of the success rates of the two treatments by stone type and overall" title="Bar chart of the success rates of the two treatments by stone type and overall" srcset="https://substackcdn.com/image/fetch/$s_!d9Ah!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png 424w, https://substackcdn.com/image/fetch/$s_!d9Ah!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png 848w, https://substackcdn.com/image/fetch/$s_!d9Ah!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png 1272w, https://substackcdn.com/image/fetch/$s_!d9Ah!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9fe19e-678a-4be0-a944-e2a1cd18826e_783x426.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Figure 1. Kidney stone treatment success rates by stone type and overall. Source: based on Charig et al. (1986) and Julious and Mullee (1994).</figcaption></figure></div><h3>2.2. Step 2: the overall rate is a weighted average</h3><p>Each option's overall rate is the average of its group rates, <strong>weighted by each group's share within that option</strong>:</p><blockquote><p>Overall rate = &#931; (rate in group i &#215; share of group i)</p></blockquote><p>Applied to kidney stones:</p><ul><li><p>A: 93.1% &#215; 25% + 73.0% &#215; 75% &#8776; <strong>78.0%</strong></p></li><li><p>B: 86.7% &#215; 77% + 68.8% &#215; 23% &#8776; <strong>82.6%</strong></p></li></ul><p>A puts 75% of its weight on the group with a low baseline rate; B puts 77% of its weight on the group with a high baseline rate. The difference in weights is large enough to swamp A's advantage within each group. This is the common mechanism behind <strong>every</strong> case of Simpson's paradox in this article.</p><h3>2.3. Three necessary conditions</h3><p>The paradox can only occur when all three conditions hold at once:</p><ol><li><p>There is a grouping variable Z (here, stone size).</p></li><li><p>Z strongly affects the outcome, meaning the groups have clearly different baseline rates.</p></li><li><p>The distribution of Z <strong>differs between the options</strong> being compared.</p></li></ol><p>Remove any one condition and the paradox cannot occur. In particular, if two options have the same group composition, their weights are identical and the aggregated figures cannot reverse. Every prevention strategy later in this article builds on this point: <strong>make the group composition the same</strong>, through study design or statistical adjustment.</p><h2>3. Simpson's paradox in practice</h2><p>The same weighting mechanism appears in many fields. The five examples below run from admissions and sports to business and epidemiology, ending with a version for continuous variables.</p><h3>3.1. Graduate admissions at Berkeley (1973)</h3><p>For the fall 1973 intake, the University of California, Berkeley was suspected of discriminating against women. The aggregated figures looked stark (Bickel et al., 1975; Table 3):</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/MesK1/2/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6aef0840-36f5-41c9-94d2-7afefd2094d1_1220x320.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c9b87ff7-18b8-4d1b-b92b-cbffc4046be3_1220x390.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 3. Graduate admission rates at Berkeley, 1973&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/MesK1/2/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>A 9-percentage-point gap across more than 12,000 applications is unlikely to be chance. But when Bickel and colleagues analysed the data <strong>department by department</strong>, most departments showed no bias against women; taken across departments, the data even leaned slightly in women's favour.</p><p>The explanation lies in where people applied. Women concentrated on <strong>highly competitive</strong> departments with low admission rates. Men concentrated on <strong>easier</strong> departments with high admission rates. The overall gap reflects <strong>the pattern of field choice</strong>, not how each department assessed applications.</p><p>The story does not end there. Why did women concentrate in the hardest-to-enter fields? The answer may lie in social factors, education and bias long before anyone applied, a point Bickel and colleagues themselves raised. The department-level analysis shows that admissions committees did <strong>not</strong> discriminate, but it does not rule out inequality elsewhere. Choosing how to split the data is therefore always an interpretive decision, not a purely technical one.</p><h3>3.2. Batting averages in baseball</h3><p>Ross (2004) illustrates the paradox with two players, Derek Jeter and David Justice, over three seasons (Table 4):</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/bBQ8l/2/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/319f5813-3aab-4994-b660-9a1cb9707058_1220x500.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/25c38915-e29b-4d8a-9213-9ca993821207_1220x570.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 4. Batting averages of Derek Jeter and David Justice&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/bBQ8l/2/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Justice beat Jeter in <strong>all three seasons</strong>, yet combined, Jeter comes out slightly ahead. Weighting is again the reason: 1995 accounted for only about 4% of Jeter's at-bats, so his worst season barely affected his combined figure. For Justice, 1995 accounted for about 39% of his at-bats, and it was also his lowest-scoring season, so his combined figure was dragged down.</p><h3>3.3. A/B testing in e-commerce</h3><p>Suppose an online store tests two checkout page designs (<em>illustrative data</em>; Table 5):</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/mKc17/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bb4c88ab-f121-43a8-afc0-2268ea638d81_1220x394.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f532e687-3ce6-45d3-be7e-b14069baad2d_1220x514.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 5. Conversion rates of two checkout designs by device (illustrative data)&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/mKc17/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>On the overall figures, B wins. But on <strong>both device types</strong>, A has the higher conversion rate. The cause: 80% of A's views came from mobile, compared with 67% of B's, and mobile users buy less to begin with. Choosing B on the overall figure means discarding the better option.</p><p>Notably, this is an <strong>experiment</strong>, not observational data. A warning sign should have been caught early: A received 1,000 views and B received 1,500, whereas an A/B test is usually designed to split traffic evenly. Traffic allocation errors like this can distort results even when randomization is in place (Fabijan et al., 2019; Kohavi et al., 2020).</p><h3>3.4. COVID-19 case fatality rates</h3><p>Early in the 2020 pandemic, <strong>Italy's case fatality rate (CFR) among recorded cases was higher than China's</strong> overall. Yet von K&#252;gelgen and colleagues (2021) showed that when split by age group, <strong>Italy's CFR was lower than China's in every age group</strong> in the data they compared.</p><p>The explanation is <strong>the age structure of recorded cases</strong>. In Italy, older people made up a much larger share of recorded cases, and this group faces a far higher risk of death. Different testing strategies in the two countries also contributed to the difference.</p><p>This example also reveals a subtlety discussed in detail below. The authors did not treat age simply as a confounder, but analysed it as a <strong>mediator</strong>: country affects who gets infected and tested, and through that, the fatality rate. Which figure is "right" depends on the question being asked. To compare quality of care, you need to compare within age groups or use <strong>age-standardized</strong> rates, the standard practice in epidemiology when comparing populations.</p><h3>3.5. A version with continuous variables</h3><p>The paradox is not limited to percentages; it also affects <strong>correlations between two continuous variables</strong>. Consider the relationship between weekly hours of exercise and cholesterol level (<em>hypothetical example</em>):</p><ul><li><p>Across all the data, the trend is <strong>upward</strong>: people who exercise more seem to have higher cholesterol.</p></li><li><p>Split by age group, each group shows a <strong>downward</strong> trend: within the same age, people who exercise more have lower cholesterol.</p></li></ul><p>The reason is that older people both exercise more (on their doctors' advice) and have higher cholesterol because of age. <strong>Age</strong> is a confounder that tilts the overall trend line upward. Figure 2 illustrates the case with simulated data. This form is easy to spot on a scatter plot <strong>coloured by group</strong>: each cloud of points slopes downward, but the line joining the centres of the clouds slopes upward.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dIRU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dIRU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png 424w, https://substackcdn.com/image/fetch/$s_!dIRU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png 848w, https://substackcdn.com/image/fetch/$s_!dIRU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png 1272w, https://substackcdn.com/image/fetch/$s_!dIRU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dIRU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png" width="787" height="480" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:480,&quot;width&quot;:787,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:52419,&quot;alt&quot;:&quot;Scatter plot of weekly exercise hours and cholesterol, coloured by age group&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Scatter plot of weekly exercise hours and cholesterol, coloured by age group" title="Scatter plot of weekly exercise hours and cholesterol, coloured by age group" srcset="https://substackcdn.com/image/fetch/$s_!dIRU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png 424w, https://substackcdn.com/image/fetch/$s_!dIRU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png 848w, https://substackcdn.com/image/fetch/$s_!dIRU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png 1272w, https://substackcdn.com/image/fetch/$s_!dIRU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd0b0e2-1d0a-4ade-bce0-5205d72316ef_787x480.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Figure 2. A downward trend within each age group becomes an upward trend when the groups are combined (simulated data).</figcaption></figure></div><h2>4. Should you trust the aggregated or the stratified figures?</h2><p>This is the central question, and the answer is surprising: <strong>the data alone cannot answer it</strong>. Two datasets can contain identical numbers yet call for opposite readings, depending on the <strong>causal structure</strong> that generated them. Pearl (2014) clarifies this using <strong>causal diagrams</strong>; epidemiologists have reached the same conclusion (Hern&#225;n et al., 2011).</p><p>Let X be the option being compared, Y the outcome, and Z the variable used to split the data. The role of Z determines which figure to read (Table 6).</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/kSJhW/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d5b445fc-2d20-4dd6-a8c4-304e42d69b9c_1220x432.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/18f3f57c-cf82-432a-8a4f-dbe214abfb50_1220x502.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 6. Three roles of the grouping variable Z&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/kSJhW/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>4.1. Case 1: Z is a confounder</h3><p>Z affects both the choice of X and the outcome Y. Example: stone size affects which treatment the doctor chooses and the chance of success. Here you <strong>should use the stratified figures</strong>, or figures adjusted for Z, because Z biases the comparison. The kidney stone and age&#8211;cholesterol examples belong to this case.</p><h3>4.2. Case 2: Z is a mediator</h3><p>Z lies on the very pathway through which X affects Y. Example: a drug lowers blood pressure, and lower blood pressure reduces the risk of stroke. If you stratify by blood pressure <em>after treatment</em>, you inadvertently block the drug's pathway, and the drug looks useless. If you want the drug's <strong>total effect</strong>, you <strong>should use the aggregated figures</strong>.</p><h3>4.3. Case 3: Z is a collider</h3><p>Z is affected by both X and Y. Controlling for Z creates a spurious association between X and Y. Example: if you look only at people <em>admitted to a prestigious university</em>, you may find test scores and athletic talent negatively correlated, even though the two are unrelated in the wider population; the university admits people who excel in one <em>or</em> the other. Here too you <strong>should use the aggregated figures</strong>.</p><h3>4.4. Bringing the three cases together</h3><p>The three cases can be condensed into one rule for choosing figures according to the role of Z (Table 7).</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/OZ76F/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/64fd2b86-5f0d-498b-abc1-cf458158e38b_1220x458.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/162331e2-30fb-40a8-8349-2e69d7468442_1220x528.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 7. Aggregated or stratified figures, by the role of Z&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/OZ76F/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>The COVID-19 example above shows that the same variable can play different roles depending on the question. The first step is therefore always to state the <strong>causal question</strong> you want to answer. For more on building and reading causal diagrams, see Greenland et al. (1999) or Pearl et al. (2016).</p><h3>4.5. A backpack campaign and a causal diagram</h3><p>Haughton et al. (2026) give a business case that shows what a causal diagram adds when the paradox appears. A company runs an advertising campaign for backpacks. Before the campaign, 2% of customers bought a backpack; six months later the figure is 2.2%, and the marketing department declares success. But split by gender, the purchase rate fell from 4% to 3.5% among men and from 1% to 0.7% among women. The overall rate rose only because the customer mix shifted towards the group that buys more anyway.</p><p>The causal diagram for this case has three variables: gender (G), the campaign (C) and backpack sales (S). Gender affects who the campaign reaches (G &#8594; C) and the demand for backpacks (G &#8594; S); the campaign affects sales (C &#8594; S). Gender is therefore a confounder, and the path through G opens a "back door" between the campaign and sales that must be blocked by stratifying or by including gender in the model. Following Case 1 above, the stratified figures are the ones to read, and they suggest the campaign did not work, or at least that there is no evidence it did. One more caveat: this is a before-and-after comparison without a control group, so a general market trend over the six months could also explain the decline.</p><p>The example extends into two principles that apply to any regression model. First, <strong>a regression coefficient has no causal meaning unless the causal structure behind it is stated</strong>. Second, one model measures one effect correctly: in a model with both campaign and gender, the campaign coefficient measures the effect of the campaign, but the gender coefficient in the same model captures only the direct effect of gender, not the part that runs through the campaign. Reading every coefficient in a regression table as a total effect is a mistake common enough that epidemiologists have named it (Westreich &amp; Greenland, 2013). Measuring several effects usually takes several models, each built from the diagram to answer its own question.</p><p>When a problem involves dozens of variables, the diagram can be drawn from domain knowledge or estimated from data with structure-learning algorithms. Either way, the conclusion is only credible if the analyst can argue that no important variable has been left out, because an unmeasured confounder still biases the result however elegant the diagram.</p><h2>5. Prevention through study design</h2><p>Treat Simpson's paradox as a <strong>default risk</strong> whenever you compare groups using aggregated figures. The response has three stages: prevention at the design stage, detection during analysis, and correct handling after detection. The first stage is the cheapest, because it breaks the third condition before the paradox can form.</p><h3>5.1. Randomization</h3><p>When patients or users are <strong>randomly</strong> assigned to options, every other variable, including ones nobody has thought of, ends up <strong>roughly evenly</strong> distributed across groups. The third condition of the paradox is removed.</p><p>Had the kidney stone study used randomization, each treatment would have received a share of large stones close to the overall share among all patients, about 49% (343/700), instead of 75% and 23% as actually happened. Only then would comparing the aggregated figures be fair.</p><p>Note: randomization only guarantees balance <em>in expectation</em>. With small samples, imbalance can still arise by chance.</p><h3>5.2. Stratified randomization</h3><p>If you know an important variable in advance (disease severity, device type, country), <strong>randomize separately within each stratum</strong>: among large-stone patients, randomly split 50/50 between A and B, and do the same among small-stone patients. This <strong>guarantees</strong> the same group composition instead of merely hoping for it. In A/B testing, it is equivalent to splitting traffic 50/50 separately for each device type.</p><h3>5.3. Record all potential confounders</h3><p>You cannot adjust for a variable you did not measure. Before collecting data, ask domain experts two questions: <em>what makes people choose option A over B</em>, and <em>what else affects the outcome</em>. Then make sure those variables are recorded (Table 8).</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/nUbfx/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0f8a7b00-e5ad-4c65-9351-40e5b1ee907e_1220x574.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b850ea6a-e70a-4fc5-9127-abd579d3ca80_1220x644.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 8. Common confounders by field&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/nUbfx/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>5.4. Check for sample ratio mismatch (SRM)</h3><p>In an A/B test, if the design is a 50/50 split but the actual data come out 53/47, that is a <strong>red flag</strong>. Technical problems such as bots, page-load errors or caching may be skewing the allocation, and Simpson's paradox may be hiding inside. Fabijan and colleagues (2019) describe sample ratio mismatch (SRM) as one of the most common and serious failures in online experiments.</p><p>The usual check is a chi-square test comparing actual counts with expected counts. If the p-value is very small (a threshold of 0.001 is common), <strong>find the cause before reading the results</strong> (Kohavi et al., 2020).</p><h2>6. Detection during data analysis</h2><p>With observational data, there is no randomization to protect you, so the analyst must actively hunt for the paradox. The tools below run from simple to sophisticated.</p><h3>6.1. Compare group composition first</h3><p>Before calculating any rate, tabulate <strong>the share of each subgroup within each option</strong>. In the kidney stone example, the composition table alone (A: 75% hard cases, B: 23% hard cases) is a strong warning sign.</p><blockquote><p><strong>Rule of thumb:</strong> if the group composition of two options differs by more than about 10 percentage points on a variable that affects the outcome, treat the aggregated figures as unreliable until you have checked them by group. This is a practical alert threshold, not a statistical standard.</p></blockquote><h3>6.2. Stratified analysis</h3><p>Split the data by each suspect variable and compare again, one variable at a time, then important combinations. The result falls into one of three cases (Table 9):</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/4nizR/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/23e1d14c-889e-4555-bd96-d103899dc9bf_1220x426.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4a610177-6b18-4bab-9ddf-52aed9b7fc14_1220x546.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 9. Three possible results when screening many variables&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/4nizR/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>Distinguishing the last case matters. If A is better for small stones but worse for large ones, the right answer is "it depends on stone size", not "A is better".</p><h3>6.3. Direct standardization</h3><p>Standardization brings the options to <strong>the same group composition</strong> for a fair comparison. For kidney stones, take all 700 patients as the standard population: 357 small-stone cases (87 + 270) and 343 large-stone cases (263 + 80). Then compute the expected successes if each treatment had treated exactly this population (Table 10):</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/74WSn/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8011826b-5c27-4f05-8bd7-cd5213a2ea62_1220x416.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/de14b721-932b-482b-94a5-522183a997de_1220x536.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 10. Direct standardization of success rates in the kidney stone example&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/74WSn/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p>After standardization, the ranking <strong>reverses and matches the within-group results</strong>. This is the figure to present to the hospital board, alongside the crude figure and an explanation of why the two differ.</p><h3>6.4. The Mantel&#8211;Haenszel odds ratio</h3><p>For data stratified into several 2&#215;2 tables, the Mantel&#8211;Haenszel method computes a single odds ratio (OR) adjusted across strata (Mantel &amp; Haenszel, 1959). For kidney stones, comparing A with B:</p><ul><li><p><strong>Crude</strong> OR: about <strong>0.75</strong>, so A looks worse than B.</p></li><li><p><strong>Mantel&#8211;Haenszel</strong> OR, adjusted for stone size: about <strong>1.45</strong>, so A is better than B.</p></li></ul><p>An OR moving from below 1 to above 1 is a <strong>quantitative signature</strong> of Simpson's paradox. When crude and adjusted estimates differ substantially (many studies use a threshold of about 10%), the variable in question is a suspected confounder. Two cautions apply. First, the OR is <em>non-collapsible</em>: crude and adjusted ORs can differ even without confounding, especially when the outcome is common, as it is here. Second, a common OR is only meaningful when the effect is reasonably homogeneous across strata; the Breslow&#8211;Day test helps check this.</p><h3>6.5. Multivariable regression and the 10% change rule</h3><p>Regression can adjust for many variables at once, including continuous ones. The screening procedure has three steps:</p><ol><li><p>Fit a model <strong>with only the option variable X</strong> and record its coefficient.</p></li><li><p>Add each suspect variable Z to the model in turn.</p></li><li><p>If X's coefficient changes by <strong>more than 10%</strong> or <strong>changes sign</strong>, Z may be a confounder and should be considered for retention.</p></li></ol><p>For kidney stones, in a logistic regression, the coefficient for B versus A is <em>positive</em> in the crude model (B looks better) but <em>negative</em> once stone size is added. The sign change confirms stone size as a confounder.</p><p><strong>Caution:</strong> the 10% rule is only a screening tool, and it is also affected by the non-collapsibility noted above. Never add a mediator or a collider to a model just because it shifts the coefficient. A variable's causal role must first be established with a causal diagram (Greenland et al., 1999).</p><h3>6.6. Systematic screening across many variables</h3><p>With high-dimensional data, repeat the following check for <strong>each grouping variable</strong> (device, country, channel, age band):</p><ol><li><p>Identify the winning option overall.</p></li><li><p>Identify the winning option within each group of that variable.</p></li><li><p>Flag variables where the option winning in every group differs from the overall winner.</p></li></ol><p>The list of flagged variables becomes the priority list for domain experts to review.</p><h3>6.7. Propensity scores and inverse weighting</h3><p>When there are too many confounders to stratify by hand, you can use the <strong>propensity score</strong>: the probability that an individual is assigned to option A, estimated from that individual's characteristics (Rosenbaum &amp; Rubin, 1983). It is used in two main ways:</p><ul><li><p><strong>Matching:</strong> for each individual in group A, find the individual in group B with the closest propensity score, then compare outcomes within pairs.</p></li><li><p><strong>Inverse probability weighting:</strong> assign weights so that the two groups have equivalent distributions of characteristics.</p></li></ul><p>After matching or weighting, <strong>check the balance</strong> of each variable before comparing outcomes; a standardized difference below 0.1 is the usual threshold (Austin, 2011).</p><p><strong>Important limitation:</strong> these techniques can only adjust for <strong>measured variables</strong>. To judge how strong an unmeasured variable would need to be to overturn the conclusion, add a <strong>sensitivity analysis</strong>, such as the E-value (VanderWeele &amp; Ding, 2017).</p><h3>6.8. Visualization</h3><p>People spot patterns in pictures faster than in tables of numbers. Three chart types are especially useful:</p><ul><li><p><strong>Grouped bar chart:</strong> one cluster of bars per subgroup, with options side by side.</p></li><li><p><strong>Scatter plot coloured by group</strong>, with a regression line for each group and one for all the data. When the group lines slope opposite to the overall line, you are looking at Simpson's paradox.</p></li><li><p><strong>100% stacked bar chart or mosaic plot:</strong> shows the group composition of each option.</p></li></ul><h2>7. Handling the paradox once detected</h2><p>Detecting the paradox is only half the job. The other half has four steps:</p><ol><li><p><strong>Draw a causal diagram</strong> to determine whether Z is a confounder, a mediator or a collider. Only when Z is a confounder should stratified or adjusted figures take priority (Pearl, 2014).</p></li><li><p><strong>Consult domain experts</strong> to test the causal assumptions. Statistics cannot replace understanding of the real-world mechanism.</p></li><li><p><strong>Report both figures</strong>, aggregated and stratified (or adjusted), with an explanation of why they differ.</p></li><li><p><strong>Choose the figure that matches the question.</strong> These two questions need two different figures:</p><ul><li><p><em>This patient has a large stone; which treatment should we use?</em> &#8594; the <strong>stratified</strong> figures.</p></li><li><p><em>What overall results does the hospital's current pattern of treatment allocation produce?</em> &#8594; the <strong>aggregated</strong> figures describe this correctly.</p></li></ul></li></ol><h3>7.1. Pitfalls in handling the paradox</h3><p>Preventing the paradox does not mean slicing the data as finely as possible (Table 11).</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/5DUvO/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/499e0031-170b-4834-a2db-706447ebc7b5_1220x968.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/06dfecda-30b7-46f9-9261-fadce62b6148_1220x1038.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 11. Pitfalls in handling Simpson's paradox&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/5DUvO/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><h3>7.2. Checklist before trusting an aggregated comparison</h3><p><strong>Design</strong></p><ul><li><p>&#9744; Was assignment to options random? If not, who decided, and on what criteria?</p></li><li><p>&#9744; Have variables that could affect both the choice of option and the outcome been recorded?</p></li><li><p>&#9744; For A/B tests: does the actual allocation match the design (SRM)?</p></li></ul><p><strong>Data</strong></p><ul><li><p>&#9744; Is the group composition (age, device, region...) comparable across options?</p></li><li><p>&#9744; Is any group heavily over-represented in one option?</p></li><li><p>&#9744; Does each subgroup have a large enough sample to draw conclusions?</p></li></ul><p><strong>Analysis</strong></p><ul><li><p>&#9744; Have results been examined within each important subgroup?</p></li><li><p>&#9744; Is the direction of the conclusion consistent overall and within each group?</p></li><li><p>&#9744; Have crude and adjusted estimates been compared?</p></li><li><p>&#9744; Were stratification variables pre-specified to avoid subgroup fishing?</p></li></ul><p><strong>Interpretation</strong></p><ul><li><p>&#9744; Has the grouping variable been identified as a confounder, mediator or collider?</p></li><li><p>&#9744; Have domain experts confirmed the causal assumptions?</p></li><li><p>&#9744; Does the report present both aggregated and stratified figures, with an explanation?</p></li><li><p>&#9744; Does the chosen figure match the decision question (for an individual or for an overall policy)?</p></li></ul><h2>8. Where Simpson's paradox hides in each field</h2><p>The paradox is not just a textbook curiosity. Almost any field that compares rates across groups with different compositions can run into it (Table 12).</p><div id="datawrapper-iframe" class="datawrapper-wrap outer" data-attrs="{&quot;url&quot;:&quot;https://datawrapper.dwcdn.net/gQclI/1/&quot;,&quot;thumbnail_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/29af904b-d841-4efa-bc1e-14b0fd223a4f_1220x840.png&quot;,&quot;thumbnail_url_full&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9f1ea086-1250-4860-945e-268f9cb5e5af_1220x910.png&quot;,&quot;height&quot;:400,&quot;title&quot;:&quot;Table 12. Where Simpson's paradox tends to appear, by field&quot;,&quot;description&quot;:&quot;&quot;,&quot;belowTheFold&quot;:true}" data-component-name="DatawrapperToDOM"><iframe id="iframe-datawrapper" class="datawrapper-iframe" src="https://datawrapper.dwcdn.net/gQclI/1/" width="730" height="400" frameborder="0" scrolling="no" loading="lazy"></iframe><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(e){if(void 0!==e.data["datawrapper-height"]){var t=document.querySelectorAll("iframe");for(var a in e.data["datawrapper-height"])for(var r=0;r<t.length;r++){if(t[r].contentWindow===e.source)t[r].style.height=e.data["datawrapper-height"][a]+"px"}}}))}();</script></div><p><strong>HR (hypothetical example).</strong> Company X finds that women's average pay is 8% lower than men's. Split by department, women's pay is equal to or higher than men's within every department; the overall gap comes from women being concentrated in lower-paying departments. The right conclusion is not "there is no problem", but that the problem lies in <strong>the allocation of roles</strong>, not in pay policy within each department. The remedy is therefore different too, echoing the lesson from Berkeley.</p><p><strong>Healthcare.</strong> When hospitals are ranked by mortality, tertiary hospitals that receive many severe cases often have higher crude rates even though their care is no worse. That is why serious rankings must <strong>adjust for case severity</strong> before comparing.</p><h2>9. Common questions</h2><p>This section briefly answers six questions that often come up when people first meet Simpson's paradox.</p><p><strong>How common is Simpson's paradox?</strong></p><p>Complete reversals are relatively rare. Pavlides and Perlman (2009) estimated that for 2&#215;2&#215;2 tables with randomly chosen cell probabilities, the chance of a reversal is only about 1 in 60. But <strong>partial distortion</strong>, where the conclusion is clearly biased without reversing, is very common in observational data.</p><p><strong>Does big data protect against the paradox?</strong></p><p>No. The problem lies in the structure of the data, not the amount. The bigger the data, the narrower the confidence intervals, and the more confident we become in a conclusion that may be wrong.</p><p><strong>So is it always safe to use the stratified figures?</strong></p><p>No. If the grouping variable is a mediator or a collider, stratifying leads to the wrong conclusion (Hern&#225;n et al., 2011; Pearl, 2014).</p><p><strong>How does Simpson's paradox differ from an interaction effect?</strong></p><p>In Simpson's paradox, the direction of the effect is <strong>the same in every group</strong> but opposite to the overall direction. In an interaction effect, the direction <strong>differs between groups</strong>, and the right answer is "it depends on the group".</p><p><strong>Do randomized experiments eliminate the paradox completely?</strong></p><p>Nearly, if they are designed and run correctly, because random allocation makes groups equivalent in expectation. However, technical faults such as SRM can still break randomization, so checks are always needed (Fabijan et al., 2019).</p><p><strong>What if I have no data to build a causal diagram?</strong></p><p>Causal diagrams are built from <strong>domain knowledge</strong>, not from data. Sketch a draft with domain experts and refine it over time.</p><h2>Conclusion</h2><p>Simpson's paradox is not a flaw in mathematics but a reminder about how people read data. An average can blend many different stories; if we do not separate them, we may reach a conclusion that is the exact opposite of reality. Yet splitting the data mechanically can also mislead. Data do not speak for themselves; our understanding of the process that generated them decides whether we read them correctly (Pearl, 2014).</p><p>Three things to remember:</p><ol><li><p><strong>Aggregating data can reverse a trend</strong> when the group composition differs between options.</p></li><li><p><strong>Whether to aggregate or stratify depends on the causal structure and the question being asked</strong>, not on the numbers.</p></li><li><p><strong>Always understand the context</strong> behind the data before drawing conclusions.</p></li></ol><p>For managers, this means not making decisions on a single summary metric such as an average KPI or an overall conversion rate. Ask for segment-level figures with an explanation of group composition, and ask <em>"what makes these groups different?"</em> before asking <em>"which option is better?"</em>.</p><p>The next time you see a report concluding that "X is better than Y", stop and ask: <strong>what is this average hiding?</strong></p><h2>References</h2><p>Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. <em>Multivariate Behavioral Research, 46</em>(3), 399&#8211;424. <a href="https://doi.org/10.1080/00273171.2011.568786">https://doi.org/10.1080/00273171.2011.568786</a></p><p>Bickel, P. J., Hammel, E. A., &amp; O'Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. <em>Science, 187</em>(4175), 398&#8211;404. <a href="https://doi.org/10.1126/science.187.4175.398">https://doi.org/10.1126/science.187.4175.398</a></p><p>Blyth, C. R. (1972). On Simpson's paradox and the sure-thing principle. <em>Journal of the American Statistical Association, 67</em>(338), 364&#8211;366. <a href="https://doi.org/10.1080/01621459.1972.10482387">https://doi.org/10.1080/01621459.1972.10482387</a></p><p>Charig, C. R., Webb, D. R., Payne, S. R., &amp; Wickham, J. E. (1986). Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. <em>British Medical Journal (Clinical Research Ed.), 292</em>(6524), 879&#8211;882. <a href="https://doi.org/10.1136/bmj.292.6524.879">https://doi.org/10.1136/bmj.292.6524.879</a></p><p>Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., &amp; Dmitriev, P. (2019). Diagnosing sample ratio mismatch in online controlled experiments: A taxonomy and rules of thumb for practitioners. In <em>Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining</em> (pp. 2156&#8211;2164). Association for Computing Machinery. <a href="https://doi.org/10.1145/3292500.3330722">https://doi.org/10.1145/3292500.3330722</a></p><p>Greenland, S., Pearl, J., &amp; Robins, J. M. (1999). Causal diagrams for epidemiologic research. <em>Epidemiology, 10</em>(1), 37&#8211;48. <a href="https://doi.org/10.1097/00001648-199901000-00008">https://doi.org/10.1097/00001648-199901000-00008</a></p><p>Haughton, D., Haughton, J., &amp; Lo, V. S. Y. (2026). <em>Cause and effect business analytics and data science</em>. CRC Press.</p><p>Hern&#225;n, M. A., Clayton, D., &amp; Keiding, N. (2011). The Simpson's paradox unraveled. <em>International Journal of Epidemiology, 40</em>(3), 780&#8211;785. <a href="https://doi.org/10.1093/ije/dyr041">https://doi.org/10.1093/ije/dyr041</a></p><p>Julious, S. A., &amp; Mullee, M. A. (1994). Confounding and Simpson's paradox. <em>BMJ, 309</em>(6967), 1480&#8211;1481. <a href="https://doi.org/10.1136/bmj.309.6967.1480">https://doi.org/10.1136/bmj.309.6967.1480</a></p><p>Kohavi, R., Tang, D., &amp; Xu, Y. (2020). <em>Trustworthy online controlled experiments: A practical guide to A/B testing</em>. Cambridge University Press. <a href="https://doi.org/10.1017/9781108653985">https://doi.org/10.1017/9781108653985</a></p><p>Mantel, N., &amp; Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. <em>Journal of the National Cancer Institute, 22</em>(4), 719&#8211;748. <a href="https://doi.org/10.1093/jnci/22.4.719">https://doi.org/10.1093/jnci/22.4.719</a></p><p>Pavlides, M. G., &amp; Perlman, M. D. (2009). How likely is Simpson's paradox? <em>The American Statistician, 63</em>(3), 226&#8211;233. <a href="https://doi.org/10.1198/tast.2009.09007">https://doi.org/10.1198/tast.2009.09007</a></p><p>Pearl, J. (2014). Understanding Simpson's paradox. <em>The American Statistician, 68</em>(1), 8&#8211;13. <a href="https://doi.org/10.1080/00031305.2014.876829">https://doi.org/10.1080/00031305.2014.876829</a></p><p>Pearl, J., Glymour, M., &amp; Jewell, N. P. (2016). <em>Causal inference in statistics: A primer</em>. Wiley. <a href="https://www.wiley.com/en-us/Causal+Inference+in+Statistics:+A+Primer-p-9781119186847">https://www.wiley.com/en-us/Causal+Inference+in+Statistics:+A+Primer-p-9781119186847</a></p><p>Pearson, K., Lee, A., &amp; Bramley-Moore, L. (1899). Mathematical contributions to the theory of evolution. VI. Genetic (reproductive) selection: Inheritance of fertility in man, and of fecundity in thoroughbred racehorses. <em>Philosophical Transactions of the Royal Society of London. Series A, 192</em>, 257&#8211;330. <a href="https://doi.org/10.1098/rsta.1899.0006">https://doi.org/10.1098/rsta.1899.0006</a></p><p>Rosenbaum, P. R., &amp; Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. <em>Biometrika, 70</em>(1), 41&#8211;55. <a href="https://doi.org/10.1093/biomet/70.1.41">https://doi.org/10.1093/biomet/70.1.41</a></p><p>Ross, K. (2004). <em>A mathematician at the ballpark: Odds and probabilities for baseball fans</em>. Pi Press. <a href="https://www.amazon.com/s?k=A+mathematician+at+the+ballpark+Ross&amp;i=stripbooks">https://www.amazon.com/s?k=A+mathematician+at+the+ballpark+Ross&amp;i=stripbooks</a></p><p>Simpson, E. H. (1951). The interpretation of interaction in contingency tables. <em>Journal of the Royal Statistical Society: Series B (Methodological), 13</em>(2), 238&#8211;241. <a href="https://doi.org/10.1111/j.2517-6161.1951.tb00088.x">https://doi.org/10.1111/j.2517-6161.1951.tb00088.x</a></p><p>VanderWeele, T. J., &amp; Ding, P. (2017). Sensitivity analysis in observational research: Introducing the E-value. <em>Annals of Internal Medicine, 167</em>(4), 268&#8211;274. <a href="https://doi.org/10.7326/M16-2607">https://doi.org/10.7326/M16-2607</a></p><p>von K&#252;gelgen, J., Gresele, L., &amp; Sch&#246;lkopf, B. (2021). Simpson's paradox in COVID-19 case fatality rates: A mediation analysis of age-related causal effects. <em>IEEE Transactions on Artificial Intelligence, 2</em>(1), 18&#8211;27. <a href="https://doi.org/10.1109/TAI.2021.3073088">https://doi.org/10.1109/TAI.2021.3073088</a></p><p>Westreich, D., &amp; Greenland, S. (2013). The Table 2 fallacy: Presenting and interpreting confounder and modifier coefficients. <em>American Journal of Epidemiology, 177</em>(4), 292&#8211;298. <a href="https://doi.org/10.1093/aje/kws412">https://doi.org/10.1093/aje/kws412</a></p><p>Yule, G. U. (1903). Notes on the theory of association of attributes in statistics. <em>Biometrika, 2</em>(2), 121&#8211;134. <a href="https://doi.org/10.1093/biomet/2.2.121">https://doi.org/10.1093/biomet/2.2.121</a></p>]]></content:encoded></item></channel></rss>