Premed · Premed · Statistics & Biostatistics
Lecture 20: Simple Linear Regression
Statistics / Biostatistics
Learning Objectives
By the end of this lecture, students will be able to:
- Describe the simple linear regression model and its assumptions
- Estimate regression coefficients using the least squares method
- Interpret the slope, intercept, and R-squared
- Perform hypothesis tests and construct confidence intervals for regression parameters
- Use residual analysis to assess model fit
Worked example (MCAT-style)
A researcher investigates the relationship between hours of weekly exercise (X) and resting heart rate (Y) in 42 healthy adults. The analysis yields: x̄ = 5.0 hours, ȳ = 72.0 bpm, S_xy = -180, S_xx = 60, and sum of squared residuals = 1200. Calculate the regression equation and interpret R².
Step 1: Calculate the slope using b₁ = S_xy / S_xx = -180 / 60 = -3.0 bpm per hour. For each additional hour of weekly exercise, resting heart rate decreases by 3.0 bpm on average.
Step 2: Calculate the intercept using b₀ = ȳ - b₁ · x̄ = 72.0 - (-3.0)(5.0) = 72.0 + 15.0 = 87.0 bpm. The regression equation is: ŷ = 87.0 - 3.0x.
Step 3: Calculate SS_Total. Since b₁ = S_xy / S_xx and the relationship between SS_Regression and slope exists, we use: SS_Regression = b₁ · S_xy = (-3.0)(-180) = 540. Then SS_Total = SS_Regression + SS_Residual = 540 + 1200 = 1740.
Step 4: Calculate R² = SS_Regression / SS_Total = 540 / 1740 = 0.31 or 31%. This means 31% of the variability in resting heart rate is explained by weekly exercise hours.
Common trap: Confusing the sign of b₁ when calculating SS_Regression—both negative values multiply to give a positive result.
Lecture Content
I. The Simple Linear Regression Model
Simple linear regression models the relationship between one continuous predictor (X) and one continuous outcome (Y). The model equation is Y = beta_0 + beta_1 * X + epsilon, where beta_0 is the y-intercept (the value of Y when X = 0), beta_1 is the slope (the change in Y for a one-unit increase in X), and epsilon is the random error term, assumed to follow N(0, sigma^2). The goal is to find the "best-fitting" straight line through the data.
Regression differs from correlation in an important way. Correlation measures association and is symmetric, with no implied directionality. Regression models a directional predictive relationship where X predicts Y, making it inherently asymmetric.
II. Least Squares Estimation
The method of least squares minimizes the sum of squared residuals: minimize sum of (y_i - y-hat_i)^2 = sum of (y_i - b_0 - b_1 x_i)^2. The estimated slope is b_1 = sum[(x_i - x-bar)(y_i - y-bar)] / sum[(x_i - x-bar)^2] = S_xy / S_xx, and the estimated intercept is b_0 = y-bar - b_1 x-bar. The fitted (predicted) values are y-hat_i = b_0 + b_1 * x_i, and the residuals are e_i = y_i - y-hat_i (observed minus predicted). An important geometric property is that the regression line always passes through the point (x-bar, y-bar).
<image>A scatterplot with a fitted regression line. Data points (x_i, y_i) are shown as dots. The regression line y-hat = b_0 + b_1*X is drawn through the data. Vertical dashed lines from several data points to the regression line represent residuals (e_i). The intercept b_0 is labeled where the line crosses the y-axis. The slope b_1 is illustrated with a "rise over run" triangle on the line. The point (x-bar, y-bar) is marked with a distinct symbol on the line.</image>
III. Interpretation of Coefficients
The slope (b_1) is interpreted as: "For each one-unit increase in X, Y changes by b_1 units on average." For example, for each additional year of age, systolic BP might increase by 0.5 mmHg. The sign of the slope indicates direction -- positive means a direct relationship, negative means an inverse one.
The intercept (b_0) is the predicted value of Y when X = 0. It may or may not be clinically meaningful, depending on whether X = 0 falls within the data range. If X is age and the intercept predicts systolic BP at age 0, this extrapolation is meaningless. More broadly, extrapolation should be avoided: do not use the model to predict Y for X values far outside the observed range.
IV. R-Squared (Coefficient of Determination)
R^2 = SS_Regression / SS_Total = 1 - SS_Residual / SS_Total. It represents the proportion of total variability in Y that is explained by the linear relationship with X. Its range is 0 to 1. In simple linear regression, R^2 equals the square of the Pearson correlation (r^2). For example, R^2 = 0.64 means 64% of the variation in Y is accounted for by X.
A high R^2 does not guarantee the model is correct -- residuals should always be checked. Conversely, a low R^2 does not mean the relationship is unimportant; clinical relevance must be assessed independently of the proportion of variance explained.
V. Inference for Regression Coefficients
The standard error of the slope is SE(b_1) = sqrt[MS_Residual / sum(x_i - x-bar)^2]. The hypothesis test for the slope tests H0: beta_1 = 0 (no linear relationship), with t = b_1 / SE(b_1), and df = n - 2. This test is equivalent to testing whether the correlation is zero. The confidence interval for the slope is b_1 +/- t_(alpha/2, n-2) * SE(b_1). The overall regression can also be tested with an F-test: F = MS_Regression / MS_Residual, with df1 = 1 and df2 = n-2. In simple linear regression, this F-test is equivalent to the t-test for the slope, since F = t^2.
VI. Assumptions of Simple Linear Regression (LINE)
The assumptions of simple linear regression can be remembered by the acronym LINE. Linearity means the relationship between X and Y is linear; this is checked by examining the scatterplot and the residuals versus fitted values plot, which should show no pattern. Independence means observations are independent of each other; this is checked through study design, and for time-ordered data, autocorrelation should be assessed. Normality means the residuals are normally distributed; this is checked with a histogram or Q-Q plot of residuals. Equal variance (homoscedasticity) means the variance of residuals is constant across all values of X; the residuals versus fitted values plot should show constant spread with no funnel shape.
<image>A four-panel residual diagnostic plot. Panel A (Residuals vs. Fitted): A random scatter of points around zero with constant spread -- assumptions met. Panel B (Residuals vs. Fitted): A funnel shape (variance increasing with fitted values) -- heteroscedasticity. Panel C (Residuals vs. Fitted): A curved pattern -- nonlinearity. Panel D (Q-Q plot of residuals): Points closely following the diagonal -- normality of residuals confirmed. Each panel is annotated with the diagnosis and recommended remedial action.</image>
VII. Prediction
Two types of intervals can be constructed at a given value X = x_0. A confidence interval for the mean response estimates E(Y | X = x_0) with uncertainty about the mean; this produces a narrower interval. A prediction interval for a new individual observation predicts a single future Y value and is wider because it includes both uncertainty about the mean and individual variability. Both intervals are narrowest at X = x-bar and widen as x_0 moves away from the center of the data.
VIII. Clinical Example
Consider the research question: is there a linear relationship between BMI (X) and fasting blood glucose (Y)? With n = 50 patients, the analysis yields b_0 = 50.2 mg/dL, b_1 = 2.8 mg/dL per BMI unit, SE(b_1) = 0.65, t = 2.8/0.65 = 4.31, df = 48, p < 0.001, and R^2 = 0.28. The interpretation is that for each 1-unit increase in BMI, fasting glucose increases by 2.8 mg/dL on average. The relationship is statistically significant (p < 0.001), and BMI explains 28% of the variability in fasting glucose. The remaining 72% is due to other factors such as genetics, diet, and medication.
<image>A complete regression analysis output figure. Panel A: Scatterplot of BMI vs. fasting glucose with the regression line, 95% confidence band (for the mean), and 95% prediction band (wider). Panel B: The residual vs. fitted values plot showing adequate assumptions. Panel C: A summary table with the regression equation, coefficient estimates, standard errors, t-statistics, p-values, and R-squared. The figure is labeled as a clinical example of BMI predicting fasting blood glucose.</image>