Goldilocks vs. Robin Hood: Using Shape-constrained Regression to Evaluate U-Shaped (Or Inverse U-Shaped) Theories in Data

Theories that predict u-shaped and inverse u-shaped relationships are ubiquitous throughout the social sciences. As a result of this widespread interest in identifying non-monotonic relationships in data and the well-known problems with standard parametric approaches based on quadratic regression models, there has been considerable recent interest in finding new ways to evaluate such theories using alternative approaches. In this paper, I propose a new semi-parametric method for evaluating these theories, which I call the "Goldilocks" algorithm. The algorithm is so named because it involves estimating three models in order to evaluate a u-shaped or inverse u-shaped hypothesis. One model is too flexible ("too hot") because it permits multiple inflection points in the expected relationship between x and y. One is too inflexible ("too cold") because it does not permit any inflection points. The final model ("just right") permits exactly one inflection point. In a simulation study based on 405 monotonic-increasing or inverse u-shaped functional forms and over 200 thousand simulated datasets, I show that my proposed algorithm outperforms the current favored method for testing u-shaped and inverse u-shaped hypotheses, which uses the Robin Hood algorithm in conjunction with a two-lines test, in terms of controlling the false rejection rate and the power of the test. I also show that these advantages of the Goldilocks algorithm can be further leveraged when it is used in an ensemble method that utilizes the output from both algorithms.