Regression Analysis Explained with Worked Examples

Regression analysis is a statistical method for understanding the relationship between variables — specifically, how changes in one or more “predictor” variables are associated with changes in an “outcome” variable. It’s one of the most widely used tools in statistics because it does two things at once: it quantifies how strong a relationship is, and it lets you predict outcomes for new data based on that relationship.

The Core Idea: Fitting a Line Through Data

Imagine you have data on hours studied and exam scores for 8 students:

Hours studied:  1   2   3   4   5   6   7   8
Exam score:    52  55  59  63  68  70  75  78

Plotted on a graph, these points roughly trend upward — more hours studied tends to associate with a higher score. Regression analysis finds the specific straight line that best fits this trend, expressed as an equation:

Score = b₀ + b₁ × (Hours studied)

Here, b₀ is the intercept (the predicted score at 0 hours studied) and b₁ is the slope (how much the score changes for each additional hour studied).

Simple Linear Regression: The Equation

The general form of a simple linear regression equation is:

ŷ = b₀ + b₁x

• ŷ (“y-hat”) — the predicted value of the outcome
• x — the predictor (independent) variable
• b₀ — the intercept, where the line crosses the y-axis
• b₁ — the slope, representing the change in y for a one-unit increase in x

For our study-hours example, running the regression calculation (typically done with software, since the formula involves several steps) produces something like:

Score = 48.5 + 3.8 × Hours

Interpreting this equation:

• Intercept (48.5): a student who studied 0 hours would be predicted to score about 48.5
• Slope (3.8): each additional hour of studying is associated with an average increase of 3.8 points

Making a prediction: for a student who studies 5 hours:

Score = 48.5 + 3.8 × 5 = 48.5 + 19 = 67.5

How the Line Is Actually Calculated: Least Squares

Regression finds the “best fit” line using a method called ordinary least squares (OLS) — it identifies the line that minimizes the total squared distance between each actual data point and the line’s predicted value at that point.

Why square the distances? Same reason as in standard deviation: squaring removes negative signs (so points above and below the line don’t cancel out) and penalizes larger errors more heavily than smaller ones, pushing the line toward a genuinely representative middle path through the data.

A Second Worked Example: Advertising Spend and Sales

Suppose a company tracks monthly advertising spend (in thousands) and resulting sales (in thousands):

Ad spend:   2    4    6    8   10
Sales:     35   50   58   72   80

Running a regression produces:

Sales = 25.4 + 5.7 × Ad spend

Interpreting it:

• Even with $0 spent on advertising, the model predicts baseline sales of about 25.4 (perhaps from repeat customers or brand recognition)
• Each additional $1,000 spent on advertising is associated with an average sales increase of $5,700

Prediction for $7,000 in ad spend:

Sales = 25.4 + 5.7 × 7 = 25.4 + 39.9 = 65.3 (thousand)

R²: How Well the Line Actually Fits

A regression equation alone doesn’t tell you how good the fit is — for that, you need R² (R-squared), which measures the proportion of variation in the outcome variable that’s explained by the predictor.

• R² = 1 — the line perfectly predicts every data point (essentially never happens with real data)
• R² = 0 — the predictor explains none of the variation in the outcome
• R² = 0.75 — the predictor explains 75% of the variation in the outcome; the remaining 25% is due to other factors not captured in the model

In the advertising example, if R² = 0.89, that means 89% of the variation in sales is explained by advertising spend alone — a strong relationship, though the remaining 11% comes from other factors (seasonality, competitor activity, pricing changes) not included in this simple model.

Correlation vs Regression: A Critical Distinction

These two concepts are closely related but answer different questions:

Critically, neither correlation nor regression proves causation — the study-hours example shows an association between hours studied and scores, but the regression itself doesn’t prove that studying causes better scores (though in this case it’s a reasonable real-world assumption). A classic counter-example: ice cream sales and drowning incidents are positively correlated, but ice cream doesn’t cause drowning — both increase in summer due to a third factor (hot weather), a case researchers call a confounding variable.

Multiple Regression: Adding More Predictors

Simple linear regression uses one predictor variable. Multiple regression extends the same idea to several predictors at once:

Score = b₀ + b₁(Hours studied) + b₂(Hours slept) + b₃(Attendance %)

This lets you account for several factors simultaneously and see each one’s individual contribution to the outcome, while holding the others constant. For example, a multiple regression might reveal that hours studied still matters even after accounting for sleep and attendance — or it might reveal that once you account for attendance, studying hours matters less than it first appeared, because attendance was actually driving both variables.

Residuals: Checking If Your Model Is Reasonable

A residual is the difference between an actual observed value and what the regression line predicted for it:

Residual = Actual value - Predicted value

For the studying example, if a student studied 4 hours and actually scored 65 (not the predicted 63.7), their residual is +1.3. Examining residuals helps check whether a linear model is actually appropriate — if residuals show a clear pattern (like a curve) rather than scattering randomly above and below zero, it’s a sign the relationship might not actually be linear, and a straight-line model may be the wrong tool for this data.

Common Student Mistakes

• Confusing correlation with causation — a strong regression relationship never proves that the predictor variable causes the outcome
• Extrapolating far beyond the data’s range — predicting outcomes for x-values well outside the range of the original data (e.g., predicting a score for 40 hours studied when the data only covered 1-8 hours) is unreliable, since the relationship might not hold outside the observed range
• Ignoring R² — reporting a regression equation without checking how well it actually fits the data can be misleading; a low R² means the equation, while mathematically valid, isn’t very useful for prediction
• Assuming the relationship must be linear — some relationships are curved (exponential, logarithmic), and forcing a straight line onto clearly curved data produces a poor and misleading fit

Frequently Asked Questions

What’s the difference between simple and multiple regression? Simple regression uses one predictor variable to predict an outcome; multiple regression uses two or more predictors simultaneously, letting you assess each one’s individual contribution while accounting for the others.

Does a high R² mean the predictor causes the outcome? No — R² measures how well the model fits the observed data, not whether a causal relationship exists. Establishing causation typically requires controlled experiments or additional evidence beyond a single regression analysis.

What does a negative slope (b₁) mean? It means the outcome variable tends to decrease as the predictor increases — for example, a regression of car value against age would typically show a negative slope, since cars tend to lose value as they get older.

Can regression be used with categorical predictors, like “yes/no” variables? Yes — categorical variables can be included using coding schemes (commonly 0/1 “dummy variables”), allowing regression models to incorporate non-numeric predictors like gender, region, or treatment group alongside numeric ones.