Premed · Premed · Statistics Biostatistics
Lecture 20: Simple Linear Regression
Statistics / Biostatistics
Learning Objectives
By the end of this lecture, students will be able to:
- Describe the simple linear regression model and its assumptions
- Estimate regression coefficients using the least squares method
- Interpret the slope, intercept, and R-squared
- Perform hypothesis tests and construct confidence intervals for regression parameters
- Use residual analysis to assess model fit
Lecture Content
I. The Simple Linear Regression Model
Simple linear regression models the relationship between one continuous predictor (X) and one continuous outcome (Y). The model equation is Y = beta_0 + beta_1 * X + epsilon, where beta_0 is the y-intercept (the value of Y when X = 0), beta_1 is the slope (the change in Y for a one-unit increase in X), and epsilon is the random error term, assumed to follow N(0, sigma^2). The goal is to find the "best-fitting" straight line through the data.
Regression differs from correlation in an important way. Correlation measures association and is symmetric, with no implied directionality. Regression models a directional predictive relationship where X predicts Y, making it inherently asymmetric.
II. Least Squares Estimation
The method of least squares minimizes the sum of squared residuals: minimize sum of (y_i - y-hat_i)^2 = sum of (y_i - b_0 - b_1 x_i)^2. The estimated slope is b_1 = sum[(x_i - x-bar)(y_i - y-bar)] / sum[(x_i - x-bar)^2] = S_xy / S_xx, and the estimated intercept is b_0 = y-bar - b_1 x-bar. The fitted (predicted) values are y-hat_i = b_0 + b_1 * x_i, and the residuals are e_i = y_i - y-hat_i (observed minus predicted). An important geometric property is that the regression line always passes through the point (x-bar, y-bar).
<image>A scatterplot with a fitted regression line. Data points (x_i, y_i) are shown as dots. The regression line y-hat = b_0 + b_1*X is drawn through the data. Vertical dashed lines from several data points to the regression line represent residuals (e_i). The intercept b_0 is labeled where the line crosses the y-axis. The slope b_1 is illustrated with a "rise over run" triangle on the line. The point (x-bar, y-bar) is marked with a distinct symbol on the line.</image>
III. Interpretation of Coefficients
The slope (b_1) is interpreted as: "For each one-unit increase in X, Y changes by b_1 units on average." For example, for each additional year of age, systolic BP might increase by 0.5 mmHg. The sign of the slope indicates direction -- positive means a direct relationship, negative means an inverse one.
The intercept (b_0) is the predicted value of Y when X = 0. It may or may not be clinically meaningful, depending on whether X = 0 falls within the data range. If X is age and the intercept predicts systolic BP at age 0, this extrapolation is meaningless. More broadly, extrapolation should be avoided: do not use the model to predict Y for X values far outside the observed range.
IV. R-Squared (Coefficient of Determination)
R^2 = SS_Regression / SS_Total = 1 - SS_Residual / SS_Total. It represents the proportion of total variability in Y that is explained by the linear relationship with X. Its range is 0 to 1. In simple linear regression, R^2 equals the square of the Pearson correlation (r^2). For example, R^2 = 0.64 means 64% of the variation in Y is accounted for by X.
A high R^2 does not guarantee the model is correct -- residuals should always be checked. Conversely, a low R^2 does not mean the relationship is unimportant; clinical relevance must be assessed independently of the proportion of variance explained.
V. Inference for Regression Coefficients
The standard error of the slope is SE(b_1) = sqrt[MS_Residual / sum(x_i - x-bar)^2]. The hypothesis test for the slope tests H0: beta_1 = 0 (no linear relationship), with t = b_1 / SE(b_1), and df = n - 2. This test is equivalent to testing whether the correlation is zero. The confidence interval for the slope is b_1 +/- t_(alpha/2, n-2) * SE(b_1). The overall regression can also be tested with an F-test: F = MS_Regression / MS_Residual, with df1 = 1 and df2 = n-2. In simple linear regression, this F-test is equivalent to the t-test for the slope, since F = t^2.
VI. Assumptions of Simple Linear Regression (LINE)
The assumptions of simple linear regression can be remembered by the acronym LINE. Linearity means the relationship between X and Y is linear; this is checked by examining the scatterplot and the residuals versus fitted values plot, which should show no pattern. Independence means observations are independent of each other; this is checked through study design, and for time-ordered data, autocorrelation should be assessed. Normality means the residuals are normally distributed; this is checked with a histogram or Q-Q plot of residuals. Equal variance (homoscedasticity) means the variance of residuals is constant across all values of X; the residuals versus fitted values plot should show constant spread with no funnel shape.
<image>A four-panel residual diagnostic plot. Panel A (Residuals vs. Fitted): A random scatter of points around zero with constant spread -- assumptions met. Panel B (Residuals vs. Fitted): A funnel shape (variance increasing with fitted values) -- heteroscedasticity. Panel C (Residuals vs. Fitted): A curved pattern -- nonlinearity. Panel D (Q-Q plot of residuals): Points closely following the diagonal -- normality of residuals confirmed. Each panel is annotated with the diagnosis and recommended remedial action.</image>
VII. Prediction
Two types of intervals can be constructed at a given value X = x_0. A confidence interval for the mean response estimates E(Y | X = x_0) with uncertainty about the mean; this produces a narrower interval. A prediction interval for a new individual observation predicts a single future Y value and is wider because it includes both uncertainty about the mean and individual variability. Both intervals are narrowest at X = x-bar and widen as x_0 moves away from the center of the data.
VIII. Clinical Example
Consider the research question: is there a linear relationship between BMI (X) and fasting blood glucose (Y)? With n = 50 patients, the analysis yields b_0 = 50.2 mg/dL, b_1 = 2.8 mg/dL per BMI unit, SE(b_1) = 0.65, t = 2.8/0.65 = 4.31, df = 48, p < 0.001, and R^2 = 0.28. The interpretation is that for each 1-unit increase in BMI, fasting glucose increases by 2.8 mg/dL on average. The relationship is statistically significant (p < 0.001), and BMI explains 28% of the variability in fasting glucose. The remaining 72% is due to other factors such as genetics, diet, and medication.
<image>A complete regression analysis output figure. Panel A: Scatterplot of BMI vs. fasting glucose with the regression line, 95% confidence band (for the mean), and 95% prediction band (wider). Panel B: The residual vs. fitted values plot showing adequate assumptions. Panel C: A summary table with the regression equation, coefficient estimates, standard errors, t-statistics, p-values, and R-squared. The figure is labeled as a clinical example of BMI predicting fasting blood glucose.</image>
