Premed · Premed · Statistics Biostatistics
Lecture 21: Multiple Regression Introduction
Statistics / Biostatistics
Learning Objectives
By the end of this lecture, students will be able to:
- Extend simple linear regression to include multiple predictors
- Interpret partial regression coefficients in a multiple regression model
- Distinguish between R-squared and adjusted R-squared
- Understand the concept of confounding and its control through regression
- Recognize collinearity and its effects on regression estimates
Lecture Content
I. From Simple to Multiple Regression
Simple linear regression uses a single predictor: Y = beta_0 + beta_1X + epsilon. Multiple linear regression extends this to include p predictors (also called independent variables or covariates): Y = beta_0 + beta_1X_1 + beta_2X_2 + ... + beta_pX_p + epsilon. Here beta_0 is the intercept (predicted Y when all X's equal 0) and each beta_j is a partial regression coefficient for X_j. Multiple regression allows predicting Y from several variables simultaneously, controlling for confounders, and assessing the independent contribution of each predictor.
II. Interpretation of Partial Regression Coefficients
Each beta_j represents the expected change in Y for a one-unit increase in X_j, holding all other predictors constant. This "holding constant" (or adjusting for) other variables is what makes multiple regression so valuable in clinical research.
For example, if Y = systolic blood pressure, X_1 = age, and X_2 = BMI, then b_1 = 0.6 means that for each additional year of age, SBP increases by 0.6 mmHg, adjusted for BMI. Similarly, b_2 = 1.2 means that for each additional BMI unit, SBP increases by 1.2 mmHg, adjusted for age. These coefficients may differ from what simple regression would yield because the adjustment removes the influence of correlated predictors. If age and BMI are correlated, the simple regression of SBP on BMI alone would inflate the BMI effect due to confounding by age.
III. Controlling for Confounding
A confounder is a variable associated with both the predictor and the outcome. In multiple regression, including a confounder as a covariate adjusts for its effect. For example, when studying the effect of exercise (X_1) on blood pressure (Y), age (X_2) is a potential confounder because older people may exercise less and have higher blood pressure. Including age in the model as Y = beta_0 + beta_1Exercise + beta_2Age + epsilon means that beta_1 now represents the effect of exercise adjusted for age. This is one of the primary uses of multiple regression in clinical research.
<image>A path diagram illustrating confounding and its control. Panel A: Three boxes labeled "Exercise" (predictor), "Blood Pressure" (outcome), and "Age" (confounder). Arrows from Age to both Exercise and Blood Pressure. A dashed arrow from Exercise to Blood Pressure labeled "Unadjusted association (confounded)." Panel B: The same diagram but with the arrow from Age to Exercise blocked (controlled), and the arrow from Exercise to Blood Pressure now labeled "Adjusted association (unconfounded, beta_1 from multiple regression)."</image>
IV. Model Fitting and the ANOVA F-Test
Coefficients are estimated by least squares, minimizing the sum of squared residuals. The overall F-test tests whether the model as a whole is significant, with H0: beta_1 = beta_2 = ... = beta_p = 0 (no predictor is useful) versus H1: at least one beta_j does not equal 0. The F-statistic is F = MS_Regression / MS_Residual, with df1 = p and df2 = n - p - 1.
Individual t-tests assess whether each coefficient is significantly different from zero: t_j = b_j / SE(b_j), with df = n - p - 1. A significant t-test means that X_j contributes to predicting Y after adjusting for all other predictors in the model.
V. R-Squared and Adjusted R-Squared
R^2 is the proportion of variance in Y explained by all predictors combined. However, R^2 always increases when more predictors are added, even if those predictors are useless, which can be misleadingly inflated in models with many predictors.
Adjusted R^2 penalizes for the number of predictors: R^2_adj = 1 - [(1 - R^2)(n - 1) / (n - p - 1)]. Unlike R^2, adjusted R^2 can decrease if a new predictor does not improve the model sufficiently, making it better suited for comparing models with different numbers of predictors. For example, a model with 3 predictors might have R^2 = 0.45 and R^2_adj = 0.42. Adding a fourth predictor might raise R^2 slightly to 0.451 while lowering R^2_adj to 0.41, indicating the fourth predictor is not helping.
VI. Types of Predictors
Continuous predictors such as age, BMI, and lab values have coefficients interpreted as change in Y per unit increase in X. Binary (dummy) variables coded as 0/1 (for example, male = 0, female = 1) have coefficients that represent the mean difference between the two groups, adjusted for other variables.
Categorical predictors with k levels require k-1 dummy variables. For example, ethnicity with 3 groups needs 2 dummy variables, with one group serving as the reference. The coefficients represent differences relative to the reference group. Interaction terms, formed as the product of two predictors (X_1 X_2), test whether the effect of one predictor depends on the level of another. For example, the effect of a drug may differ by sex, which would be captured by including a Drug Sex interaction term.
<image>A coefficient plot (dot-and-whisker plot) from a multiple regression model predicting systolic blood pressure. Each row shows a predictor: Age, BMI, Smoking (yes/no), Sex (female vs. male reference), Exercise (hours/week). For each, a dot represents the estimated coefficient and a horizontal line represents the 95% CI. A vertical dashed line at zero marks no effect. Predictors whose CIs exclude zero are highlighted as significant. The adjusted R-squared is displayed at the bottom.</image>
VII. Multicollinearity
Multicollinearity occurs when two or more predictors are highly correlated with each other. It causes several problems: inflated standard errors of coefficients (producing wide CIs and non-significant p-values), unstable estimates (where small changes in data cause large changes in coefficients), and difficulty isolating the independent effect of each predictor.
Detection methods include examining the correlation matrix for |r| > 0.7 between predictors and computing the Variance Inflation Factor (VIF): VIF_j = 1 / (1 - R_j^2), where R_j^2 is the R-squared from regressing X_j on all other predictors. A VIF exceeding 5 or 10 suggests problematic collinearity. Remedies include removing one of the correlated predictors, combining correlated predictors into a composite score, or using principal components regression or ridge regression.
VIII. Model Building Strategies
The preferred approach is theory-driven model building, where predictors are included based on clinical knowledge and prior research. Automated methods such as forward selection (adding predictors one at a time), backward elimination (starting with all predictors and removing the least significant iteratively), and stepwise selection (combining forward and backward) should be used cautiously because they are prone to overfitting, produce unstable results, and can inflate significance.
Model diagnostics should always include checking residual plots, identifying influential observations, and assessing multicollinearity. The parsimony principle states that simpler models that explain the data adequately should be preferred over more complex ones.
IX. Assumptions (Same as Simple Regression, Extended)
The assumptions of multiple regression mirror those of simple regression. Linearity requires that Y is linearly related to each predictor, which can be checked with partial residual plots. Independence requires that observations are independent. Normality requires that residuals are normally distributed. Equal variance requires that residuals have constant variance (homoscedasticity). No perfect multicollinearity requires that no predictor is a perfect linear function of others. When assumptions are violated, transformations, robust methods, or alternative models (such as logistic regression for binary outcomes) may be needed.
<image>A residual diagnostic panel for a multiple regression model. Panel A: Residuals vs. fitted values showing random scatter (good). Panel B: Q-Q plot of residuals showing approximate normality. Panel C: Scale-location plot (sqrt of standardized residuals vs. fitted values) showing constant spread. Panel D: A VIF bar chart for all predictors, with a horizontal dashed line at VIF = 5 marking the concern threshold. One predictor exceeds this threshold, highlighted in red with an annotation recommending investigation.</image>
