
Key takeaways
"Regression" comes from Francis Galton's observation that children's heights "regress" towards the average; regression to the mean is still a common thinking trap.
In statistics, regression is a family of techniques (linear, logistic, Poisson…). In machine learning, it is a type of task: predicting a continuous output, as opposed to classification.
Logistic regression is a regression technique in statistics but usually a classification algorithm in machine learning.
Explaining and predicting are different jobs. Explanatory models care about coefficients and need a proper research design before anyone talks about causation; predictive models care about error on data they have never seen.
A flexible model can fit training data perfectly and still predict worse than a simple line: always judge a predictive model on a holdout set.
"What will revenue be next quarter?" "Which factors really drive purchase decisions?" "How likely is this customer to default?" Questions like these all lead to a familiar tool: regression.
But "regression" is used in two senses. When statisticians say "regression", they mean a family of techniques. When machine-learning engineers say it, they mean a type of task. The two uses share a mathematical foundation but serve different purposes: explaining a mechanism or predicting something unknown. Mixing up these purposes causes many mistakes in business analysis, from reading a coefficient as if it were a causal effect to showing off an "accurate" model only because it fits the data used to build it.
This article clarifies the difference. Section 1 tells the story of where the word "regression" came from. Section 2 separates the two uses of the term in statistics and machine learning. Section 3 introduces two basic regression techniques, linear and logistic regression, and Section 4 describes the regression task in machine learning. Section 5 is the heart of the article: how models built to explain differ from models built to predict, in the question they answer, how they are built and how they are evaluated, with a short Python example. Section 6 shows how to read the evaluation of a regression model correctly.
1. Where does the word "regression" come from?
In the late nineteenth century, Francis Galton measured the heights of parents and their children. He noticed that very tall parents tended to have tall children, but less tall than themselves; very short parents had short children, but less short. The next generation's height tended to "go back" towards the average. Galton (1886) called this regression towards mediocrity. The method he later used to draw a line describing the relationship between two variables took the name "regression", even though today it is used for all kinds of relationships, not only the "return to the average".
Regression to the mean is still a common thinking trap. A salesperson who has an unusually strong month usually has a weaker month next, even though nobody did anything wrong. If the company happened to "reprimand" that person in between, it is easy to conclude, wrongly, that reprimands don't work while praise "spoils" staff.
2. One word, two uses
Today statisticians and machine-learning engineers both say "regression" but often do not mean the same thing. Table 1 puts the two uses side by side.
The third row of Table 1 is the most confusing point: logistic regression has "regression" in its name but, in machine learning, is a classification algorithm. Conversely, the regression task in machine learning uses not only linear regression but also decision trees, random forests, gradient boosting, neural networks and more; many of these algorithms have versions for both classification and regression.
3. Regression techniques in statistics
Statistics offers many regression techniques, each suited to a type of dependent variable (James et al., 2021). This section introduces the two most basic and widely used in business analysis: linear regression for continuous variables and logistic regression for binary ones.
3.1. Linear regression
Linear regression assumes that the dependent variable y (a continuous variable such as revenue, price or temperature) changes along a straight line with the independent variables x₁, x₂, …:
y = β₀ + β₁x₁ + β₂x₂ + … + ε
where β₀ is the intercept, β₁, β₂… are the coefficients of each variable, and ε is the error the model cannot explain. The coefficients are usually estimated by ordinary least squares (OLS): choosing the line that minimises the sum of squared distances from the data points to the line. Figure 1 illustrates the idea with simulated data on advertising budget and sales: each point is a month, the red line is the estimated regression line, and the dashed segment is the error for one particular month.

Example (illustrative figures). A marketing director regresses monthly sales on the digital advertising budget and gets:
Sales (thousand USD) = 500 + 10 × Advertising budget (thousand USD)
How to read it: with a zero budget, the model estimates sales of about USD 500,000; each additional USD 1,000 of advertising is associated with about USD 10,000 more in sales, holding other factors constant. Note the word "associated": if the company usually raises advertising exactly in the peak season, the coefficient of 10 may also be measuring the effect of seasonality (see Section 5).
3.2. Logistic regression
When the dependent variable is binary (yes/no, default/no default, 1/0), linear regression can produce "probabilities" below 0 or above 1. Logistic regression solves this by passing the linear combination through the sigmoid function, which keeps the result between 0 and 1:
P(y = 1) = 1 / (1 + e^−(β₀ + β₁x₁ + …))
Example (illustrative). A bank predicts the probability of default from income, current debt and credit history. The model does not answer "yes" or "no"; it gives a probability, say 0.85. The bank then chooses a threshold for its decision: a low threshold rejects many good customers, a high one accepts many bad ones. Choosing the threshold is a business decision, not the model's.
4. The regression task in machine learning
In machine learning, a regression task is a supervised learning task whose output is a continuous number: a house price, a delivery time, electricity consumption. When the data is large and the relationships are complex and non-linear, practitioners use algorithms such as regression trees, random forests, gradient boosting (XGBoost, LightGBM), support vector regression (SVR) or neural networks.
Example. Ride-hailing and food-delivery apps such as Uber and Grab, Southeast Asia's leading super-app, have to estimate the estimated time of arrival (ETA): the output is a number of minutes, so this is a regression task. The model uses many features, such as distance, time of day, traffic and weather. Surge pricing is more complex: it relies on forecasts of supply and demand by area (which may use regression models) plus rules and price-optimisation algorithms.
5. Explain or predict?
Shmueli (2010) showed that modelling to explain and modelling to predict are two different activities, even when they use the same equation. Breiman (2001) called this the "two cultures" of statistical modelling. Table 2 compares the two purposes on five aspects; the four subsections that follow cover each purpose, how to evaluate models and a worked example.
5.1. Regression to explain: correlation is not causation
When regression is used to explain, researchers care about the coefficients and how reliable they are. But a regression equation by itself only measures how variables move together; it does not prove that one variable causes another.
The classic example: ice-cream sales are positively correlated with drownings. Selling ice cream does not cause drowning; both rise with summer temperatures, a confounder. Leaving out a confounder biases the regression coefficients; putting the wrong variable in can bias them too, as Simpson's paradox shows.
To conclude anything about causation, you need the right design: randomised controlled trials (RCTs), A/B tests, or causal inference methods such as difference-in-differences, instrumental variables and regression discontinuity. Regression is often still the calculating engine inside those designs, but it is the design that allows causal claims.
5.2. Regression to predict: estimating the unknown
"Prediction" is not just "forecasting the future". Prediction means using known data to estimate an unknown value, at any point in time:
The past: palaeontologists estimate the body mass of dinosaurs from the circumference of the femur and humerus, based on regression relationships measured in living animals (Campione & Evans, 2012).
The present: Zillow's Zestimate estimates the current price of a home that is not for sale. Central banks use nowcasting to estimate GDP growth for the current quarter from data released earlier (industrial output, retail sales, trade).
The future: forecasting next month's sales or tomorrow's electricity demand. With stock prices, experience shows that models predicting short-term prices rarely beat a naive forecast consistently, a reminder that not everything can be predicted.
In prediction tasks, the key question is how wrong the model is on new data. People using the Zestimate need a figure close to the real price; they care little about how many percentage points a red tiled roof adds.
5.3. Evaluating models: all the data or a holdout?
Because the two purposes differ, so does evaluation, especially on the question of which data to use to measure quality.
Explanatory models are usually estimated on the whole sample: a larger sample gives more precise coefficient estimates (smaller standard errors). Quality is judged by checking model assumptions, the meaning of the coefficients and their fit with theory.
Predictive models must not be evaluated on the data used to train them. A sufficiently complex model can "memorise" even the noise, a problem called overfitting: the error on training data is tiny but the error on new data is large. The data is therefore split into a training set and a holdout (test) set, or cross-validation is used. With time-series data, the test set must come after the training set, never shuffled in at random.
Table 3 summarises three common metrics for regression tasks.
5.4. A short Python example
The code below uses the same simulated data as Figure 1, splits it into a training set and a test set, and compares the errors of linear regression on the two sets.
```python import numpy as np from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
rng = np.random.default_rng(42) budget = rng.uniform(10, 100, 200).reshape(-1, 1) # thousand USD sales = 500 + 10 * budget.ravel() + rng.normal(0, 80, 200) # illustrative
X_train, X_test, y_train, y_test = train_test_split( budget, sales, test_size=0.25, random_state=0)
model = LinearRegression().fit(X_train, y_train) print("Intercept:", round(model.intercept_, 1), "| Coefficient:", round(model.coef_[0], 2))
for name, X, y in [("Train", X_train, y_train), ("Test", X_test, y_test)]: pred = model.predict(X) print(name, "R2 =", round(r2_score(y, pred), 3), "| MAE =", round(mean_absolute_error(y, pred), 1), "| RMSE =", round(mean_squared_error(y, pred) ** 0.5, 1)) ```
The estimated coefficients are fairly close to 500 and 10 (about 469 and 10.55) because the data was generated from that very equation plus noise; the gap comes from the noise and the small sample. Table 4 compares the results of linear regression with a very flexible model, a decision tree with unlimited depth (DecisionTreeRegressor()).
Table 4 shows that the decision tree "memorises" the training set completely (zero error) but predicts much worse on new data, while linear regression has similar errors on both sets. Figure 2 visualises the RMSE column of Table 4: the gap between the decision tree's two bars is the signature of overfitting.

6. Reading a regression evaluation correctly
Table 3 in Section 5.3 introduced the evaluation metrics. This section uses a small example to show how they differ, separates two kinds of interval that are often confused, and lists three biases that let a model score well on paper and still fail in use.
6.1. Five days, four metrics
A model forecasts the number of coffees a shop sells each day; on five days of the test set the actual counts are 120, 150, 90, 200 and 140 cups, and the forecasts are 110, 160, 100, 170 and 150 (illustrative figures). Figure 3 sets the two series side by side with each day's error.

From the five errors in Figure 3: MAE (mean absolute error) is (10 + 10 + 10 + 30 + 10) ÷ 5 = 14 cups. RMSE (root mean squared error) is the square root of (100 + 100 + 100 + 900 + 100) ÷ 5 = 260, about 16.1 cups; RMSE exceeds MAE because, once squared, the 30-cup error on day four weighs nine times as much as each 10-cup error. MAPE (mean absolute percentage error) is the average of 8.3%, 6.7%, 11.1%, 15.0% and 7.1%, about 9.7%. R² is 1 minus the sum of squared errors (1,300) divided by the sum of squared deviations from the mean of 140 cups (6,600), about 0.80: compared with guessing 140 cups every day, the model removes about 80% of the squared error.
Choose the metric by the cost of mistakes. If running out of supplies on one day costs far more than a few days with slight surpluses, large errors are what matter and RMSE reflects that better than MAE. For a store manager, MAE ("off by 14 cups a day on average") is easiest to grasp. To compare accuracy across hundreds of items with very different volumes, MAPE gives a common scale, except for items with zero-sale days. It also pays to look at the signed mean error: here (10 − 10 − 10 + 30 − 10) ÷ 5 = 2 cups, so the model hardly leans to one side. And every number needs a benchmark: an MAE of 14 cups means something only next to a simple rule such as last week's sales on the same weekday. Two metrics can even pick different models: on a used-car price dataset, one method had the lower MAE but the higher RMSE, because it missed one car very badly (Camm et al., 2024).
6.2. A confidence interval is not a prediction interval
A regression model produces two kinds of interval, and reports often confuse them. Camm et al. (2024) use the example of a trucking company predicting that a 105-mile delivery with three stops takes 9.25 hours. The 95% confidence interval for the mean time of all such trips is only about 0.19 hours either side (9.06 to 9.44 hours). The 95% prediction interval for one particular trip is about 1.65 hours either side (7.61 to 10.90 hours), nearly nine times as wide, because besides the error in estimating the regression line there is the trip's own variability. A dispatcher scheduling this trip must look at the prediction interval. Both intervals widen as inputs move away from the range of the training data, so every prediction should come with an interval and a note of the range the data covered.
6.3. Three biases that flatter evaluation scores
Splitting the data correctly and choosing the right metric are still not enough. Table 5 lists three biases that can let a model score well on the test set and still fail in use.
Shortcut learning in Table 5 is well documented in machine-learning research (Geirhos et al., 2020). All three biases lead to one principle: an evaluation score is a conditional promise, valid only for the kind of data used to measure it.
Conclusion
The word "regression" was born from Galton's observation of heights "returning to the average" and became the name of both a family of statistical techniques and a type of machine-learning task. Understanding the two uses avoids common confusions, such as calling logistic regression a regression task, or assuming that every numerical prediction problem calls for linear regression.
More importantly, every time analysts run a regression model, they should ask themselves whether they are explaining or predicting. When explaining, the coefficients only show how variables move together, and causal claims need a proper research design. When predicting, the only trustworthy yardstick is the error on data the model has never seen. Two different questions demand two different ways of building and evaluating a model, even when the equation is identical.
References
Breiman, L. (2001). Statistical modeling: The two cultures. Statistical Science, 16(3), 199–231. https://doi.org/10.1214/ss/1009213726
Camm, J. D., Cochran, J. J., Fry, M. J., & Ohlmann, J. W. (2024). Business analytics (5th ed.). Cengage Learning. https://www.cengage.com/c/business-analytics-5e-camm-cochran-fry-ohlmann/9780357902202/
Campione, N. E., & Evans, D. C. (2012). A universal scaling relationship between body mass and proximal limb bone dimensions in quadrupedal terrestrial tetrapods. BMC Biology, 10, 60. https://doi.org/10.1186/1741-7007-10-60
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021). AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7), 610–619. https://doi.org/10.1038/s42256-021-00338-7
Galton, F. (1886). Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246–263. https://doi.org/10.2307/2841583
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665–673. https://doi.org/10.1038/s42256-020-00257-z
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning (2nd ed.). Springer. https://www.statlearning.com/
Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289–310. https://doi.org/10.1214/10-STS330

