8 Panel Data Models
Controlling for Unobserved Heterogeneity with Repeated Observations
Panel data tracks the same units over time, letting us control for unobserved time-invariant heterogeneity. This chapter covers the full progression: pooled OLS, fixed effects (LSDV and within estimator), random effects (GLS), and the Hausman test for choosing between them. Sub-pages develop the visual intuition for each estimator.
This page is the hub for Chapter 15. It covers notation, pooled OLS, the summary comparison of all three estimators, and the decision flowchart. The detailed treatments live on dedicated sub-pages:
- Fixed Effects: The Intuition – omitted variable bias from group heterogeneity, demeaning, the within estimator, and what FE cannot estimate.
- Random Effects: The Intuition – the error components model, the \(\hat{\alpha}\) spectrum from OLS to FE, GLS partial demeaning, and the Hausman test.
- Cluster-Robust Standard Errors – why OLS standard errors are wrong with grouped data, intraclass correlation, and when to cluster.
- Dynamic Panels – lagged dependent variables, Nickell bias, and Arellano-Bond GMM.
- Hausman-Taylor – estimating time-invariant effects (like returns to education) without external instruments.
You should be comfortable with OLS, omitted variable bias, and the idea of endogeneity before reading this chapter. Familiarity with instrumental variables helps for understanding the Hausman test.
So far, we’ve worked with data where each observation appears once – a cross-section. But some of the most powerful analyses in economics come from watching the same people, firms, or countries over time. That’s panel data.
8.1 What Is Panel Data?
A panel dataset has two dimensions:
- Cross-sectional: \(N\) different units (individuals, firms, countries)
- Time-series: \(T\) time periods for each unit
So we observe \(y_{it}\) and \(x_{it}\) – the outcome and regressors for individual \(i\) at time \(t\).
Some examples to anchor the idea:
- 500 workers observed for 5 years each: \(N = 500\), \(T = 5\), total = 2,500 observations
- 50 countries observed quarterly for 10 years: \(N = 50\), \(T = 40\), total = 2,000 observations
- 1,000 firms observed annually for 3 years: \(N = 1000\), \(T = 3\), total = 3,000 observations
Why bother? Because you can see how the same unit evolves over time. That lets you control for things you can’t measure – like ability, firm culture, or managerial quality – as long as those things don’t change over time. We’ll see exactly how this works in Fixed Effects.
8.1.1 Balanced vs. Unbalanced Panels
A balanced panel means every individual is observed in every time period. An unbalanced panel means some individuals are missing in some periods – they dropped out of the survey, the firm went bankrupt, or they just didn’t respond.
Unbalanced panels are more realistic. Three common reasons:
- Attrition: People leave surveys (they move, refuse to continue, or die)
- Entry/exit: Firms enter or leave markets
- Non-response: Missing data for some periods
Modern software handles both. But always check whether missingness is random or systematic – if all the low-wage workers drop out, your estimates are biased.
8.2 Pooled OLS: The Naive Approach
The simplest thing you can do with panel data is ignore the panel structure entirely. Just stack all the observations and run OLS:
\[ y_{it} = \beta_1 + \beta_2 x_{2it} + \beta_3 x_{3it} + e_{it} \tag{8.1}\]
A note on notation: in the pooled model, \(e_{it}\) is the full error term – it includes any individual-specific effects we haven’t modeled. Later, when we decompose the error into \(u_i + e_{it}\), the symbol \(e_{it}\) refers specifically to the idiosyncratic component (the part that varies across both individuals and time).
Notice: the coefficients have no \(i\) or \(t\) subscripts. We’re assuming everyone has the same intercept and the same slopes.
For this to work, we need:
- The relationship between \(x\) and \(y\) is the same across all units
- The relationship is the same across all time periods
- No important omitted time-invariant variables
- Errors are uncorrelated within individuals across time
These assumptions almost never hold in practice. Individuals differ in important ways that pooled OLS can’t capture. If those differences are correlated with the regressors, you get omitted variable bias – exactly the same problem we dealt with using instrumental variables.
Even when pooled OLS is consistent, the standard errors are wrong. Observations on the same individual over time are correlated, and the usual OLS standard errors assume all observations are independent. The fix is cluster-robust standard errors. See Cluster-Robust Standard Errors for the full treatment.
Regular OLS treats each observation as independent information. But if Elena’s wages are positively correlated over time, then five years of data on Elena contains less independent information than five observations on five different people. Cluster-robust SEs account for this – the effective sample size is smaller than the raw observation count suggests, so the standard errors increase.
8.3 Fixed Effects in Brief
The fixed effects model allows each individual to have their own intercept:
\[ y_{it} = \beta_{1i} + \beta_2 x_{2it} + \beta_3 x_{3it} + e_{it} \tag{8.2}\]
The slopes are the same for everyone, but each person gets their own baseline level \(\beta_{1i}\). All the individual-specific, time-invariant stuff – ability, family background, firm culture – gets absorbed into that intercept.
The within estimator achieves this by demeaning: subtracting each individual’s time average from every observation. Because \(\beta_{1i}\) is constant over time, it cancels out. What remains is a regression on within-individual deviations – only variation within a person over time identifies the slope. The trade-off: any variable that doesn’t change over time (race, gender, country of origin) also gets wiped out and cannot be estimated.
You can also include time fixed effects \(\lambda_t\) to absorb common shocks (recessions, policy changes) that affect all units simultaneously. Most applied papers include both individual and time fixed effects.
For the full visual walk-through – omitted variable bias from group heterogeneity, class imbalance, and the demeaning derivation – see Fixed Effects: The Intuition.
Two reasons. First, FE can’t estimate the effect of any time-invariant variable – race, gender, country of origin, anything that doesn’t change over time. If those coefficients are what you care about, FE is out. Second, FE throws away all between-unit variation and uses only within-unit variation, which means larger standard errors. If the RE assumptions hold, RE is more efficient.
8.4 Random Effects in Brief
Fixed effects treats each \(\beta_{1i}\) as a fixed parameter. Random effects instead decomposes it into a population mean plus a random deviation:
\[ y_{it} = \bar{\beta}_1 + \beta_2 x_{2it} + \beta_3 x_{3it} + (u_i + e_{it}) \tag{8.3}\]
The composite error \(v_{it} = u_i + e_{it}\) has two parts: an individual-specific component \(u_i\) (constant over time) and an idiosyncratic component \(e_{it}\). RE uses GLS to account for the within-individual correlation this creates, partially demeaning the data by a fraction \(\hat{\alpha}\) that interpolates between pooled OLS (\(\hat{\alpha} = 0\)) and fixed effects (\(\hat{\alpha} = 1\)).
The critical assumption is \(\text{Cov}(u_i, x_{kit}) = 0\): the individual effect must be uncorrelated with the regressors. When this holds, RE is more efficient than FE and can estimate time-invariant effects. When it fails, RE is inconsistent.
For the error components decomposition, the \(\hat{\alpha}\) spectrum, GLS partial demeaning, and the Hausman test, see Random Effects: The Intuition.
8.5 Panel Data Model Summary
Here’s how all the pieces fit together:
| Model | Assumes | Estimates time-invariant effects? | Consistency requires |
|---|---|---|---|
| Pooled OLS | No individual heterogeneity | Yes | No correlation between \(x\) and omitted effects |
| Fixed Effects | Individual heterogeneity present | No | Strict exogeneity: \(E(e_{it} \mid \mathbf{x}_i, \alpha_i) = 0\) |
| Random Effects | Individual heterogeneity present | Yes | \(\text{Cov}(u_i, x) = 0\) |
Both FE and RE require strict exogeneity: \(E(e_{it} \mid x_{i1}, \ldots, x_{iT}, \alpha_i) = 0\). This means current errors must be uncorrelated with past, present, and future values of \(x\). Two important things this rules out: (1) lagged dependent variables as regressors – if \(y_{i,t-1}\) appears on the right-hand side, FE is inconsistent for small \(T\) (this is called Nickell bias), and (2) feedback effects where past \(y\) affects future \(x\) (e.g., past earnings affecting future education decisions). When strict exogeneity fails, you need dynamic panel methods.
8.6 Choosing Between FE and RE
The Hausman test compares FE and RE estimates. Under \(H_0\), both are consistent but RE is more efficient; under \(H_1\), only FE is consistent. A large difference between the two estimates (relative to the difference in variances) means the RE assumption fails.
The Mundlak test offers an equivalent approach that works seamlessly with cluster-robust standard errors: add the individual means of the time-varying regressors to the RE model and test their joint significance. If significant, the individual effects are correlated with regressors – use FE.
Both tests are covered in detail on the Random Effects sub-page.
8.6.1 The Decision Tree
- Test for individual effects (F-test or LM test). If none, use pooled OLS.
- If effects exist, run both FE and RE. Compare with the Hausman test.
- If Hausman rejects: Use FE. The individual effects are correlated with regressors.
- If Hausman fails to reject: Use RE. It’s more efficient and can estimate time-invariant effects.
- Always use cluster-robust standard errors regardless of which model you choose.
Not necessarily. The Hausman test has low power in small samples, so “fail to reject” doesn’t prove RE is correct – it just means you couldn’t detect a difference. Many practitioners would still use FE for robustness, especially if there’s economic reason to suspect that individual effects are correlated with the regressors. A p-value of 0.08 is not reassuring.
8.7 Beyond FE and RE
Panel data methods extend well beyond the basic FE/RE choice:
Cluster-robust standard errors correct inference when within-group correlation makes OLS standard errors too small. Essential regardless of whether you use pooled OLS, FE, or RE. See Cluster-Robust Standard Errors.
Dynamic panels arise when the lagged outcome \(y_{i,t-1}\) appears on the right-hand side. FE is inconsistent in this case (Nickell bias), and Arellano-Bond GMM uses lagged levels as instruments. See Dynamic Panels.
Hausman-Taylor estimation occupies the middle ground between FE and RE: it recovers coefficients on time-invariant variables (like education or gender) without requiring external instruments, by using within-group deviations of time-varying exogenous regressors as instruments. See Hausman-Taylor.
Seemingly Unrelated Regressions (SUR) applies when \(T\) is large relative to \(N\) (a long, narrow panel). Each unit gets its own equation with different coefficients, and joint GLS estimation exploits contemporaneous error correlation for efficiency. This is Zellner’s method – see the textbook treatment in Chapter 15.
Start with Fixed Effects: The Intuition for a visual walk-through of why demeaning works, then move to Random Effects: The Intuition for the \(\hat{\alpha}\) spectrum and the Hausman test. For advanced topics, continue to Cluster-Robust SEs, Dynamic Panels, and Hausman-Taylor.
For the chapter on qualitative and limited dependent variable models, see Ch. 16. To review instrumental variables (the same Hausman test logic), see Instrumental Variables.