Are we cautious enough when using linear models? After the 1970s linear models became the most common method for quantitative social scientists. More discussion on their scope and limitations is needed. We focus on one stage of the modeling process, namely, variable selection. We show that a rigorous comparison between bivariate and multivariate regression models should be done in this stage as non-orthogonality among predictors can lead to ambiguous estimates. Further, we use geometrical representations of linear models for two purposes. First, to visualize sources of instability and the causes of ambiguous results. Second, to support residual regression as an alternate approach. We illustrate our ideas using data collected by Cukierman et al. (2002) on the relationship between government regulation and inflation in 26 countries. Our conclusions stress the need to assess structural effects and support parsimonious models with few predictors.