School Rankings Based on Spending and Outcomes Suffer from Severe Methodological Flaws and Insufficient Research

Share Article

Key New NEPC Review Takeaway: Analyses of the performance of state education systems provide misguided and unfounded conclusions.

The Reason Foundation recently published a policy brief that offers an alternative ranking of states’ education systems. The brief, which was based on a working paper from the Department of Finance and Managerial Economics at the University of Texas at Dallas, purports to offer needed adjustments and nuance, but makes its own serious mistakes, according to a new review.

Rutgers professor Bruce D. Baker reviewed Everything You Know About State Education Rankings Is Wrong and the underlying working paper, Fixing the Currently Biased State K-12 Education Rankings. He found the analyses provided did little or nothing to advance the conversation about the effectiveness of state education systems.

The twin reports begin with the presumption that high average test scores combined with lower school spending should be the basis for state rankings, which are reasonable premises, depending upon how the analyses are approached. But the reports then head off the rails, Professor Baker explains.

Offering a ‘corrected’ representation of student outcomes and a crude analysis asserting that spending has no relation to those outcomes, the reports declare states such as New Jersey and Vermont to be poor-performing, highly inefficient systems by comparison to many states. The reports then estimate a regression model and assert that the higher performing states are those with (a) weaker teachers’ unions and (b) more children in charter schools.

However, Baker’s review details how the reports’ so-called corrections involved unreasonable and illogical assumptions and adjustments. For example, the reports re-weight racial and ethnic subgroups so that they inappropriately place equal weight in states like Vermont or Wyoming on students comprising 1 to 2% of the population as the other 98 to 99%. Other problems concern a decision to ignore economic status entirely and a poorly executed adjustment for cost of living.

Regressing multiple, highly related, interdependent measures against a specious outcome measure leads to even more suspect findings and, Baker concludes, would only mislead policymakers.

Find the review, by Bruce D. Baker, at:
http://nepc.colorado.edu/thinktank/review-rankings

Find Everything You Know About State Education Rankings Is Wrong, written by Stan J. Liebowitz and Matthew L. Kelly and published by the Reason Foundation, at:
https://reason.com/archives/2018/10/07/everything-you-know-about-stat

Find Fixing the Currently Biased State K-12 Education Rankings, written by Stan J. Liebowitz and Matthew L. Kelly and published by the Department of Finance and Managerial Economics at the University of Texas at Dallas at:
https://ssrn.com/abstract=3185152

Find Documents:
Publication Announcement: nepc.info/node/9518
NEPC Review: nepc.colorado.edu/thinktank/review-rankings
Report Reviewed: reason.com/archives/2018/10/07/everything-you-know-about-stat

NEPC Reviews (http://thinktankreview.org) provide the public, policymakers, and the press with timely, academically sound reviews of selected publications. NEPC Reviews are made possible in part by support provided by the Great Lakes Center for Education Research and Practice: http://www.greatlakescenter.org

The National Education Policy Center (NEPC), housed at the University of Colorado Boulder School of Education, produces and disseminates high-quality, peer-reviewed research to inform education policy discussions. Visit us at: https://nepc.colorado.edu

Share article on social media or email:

View article via:

Pdf Print

Contact Author

William J. Mathis

Bruce D. Baker
Rutgers, The State University of New Jersey
(732) 932-7496
Email >
@NEPCtweet
since: 09/2010
Follow >