Money Magazine Poll Critique
U.VA. Professor Challenges Credibility of Money Magazine's "Best Places To Live In America"
CHARLOTTESVILLE, VA, June 11 -- Money magazine's widely publicized annual survey, "Best Places to Live in America, " is seriously flawed and misleading, according to a forthcoming analysis of the poll.
Thomas M. Guterbock, director of the University of Virginia's Center for Survey Research, claims that the Best Places report uses inadequate statistical methods and "substantially misrepresents the public's views."
In a study scheduled to appear in Public Opinion Quarterly this August, Guterbock examines problems with the report, which ranks the nation's 300 largest metropolitan areas from best to worst. The annual survey has become something of an institution despite criticism of its widely varying results from year to year.
Guterbock, a U.Va associate professor of sociology, says that the poll's "disturbing degree of volatility, "with some cities climbing or falling more than 100 places in the ranking in one year, undermines the credibility of survey research. As an example, he points to Detroit that was ranked number 295 in 1994 and rose to number 56 in 1995. On the other hand, Benton Harbor, Mich., plummeted from number 47 in 1995 to 249 in 1996.
Guterbock's research is based on Money magazine's reports of its statistical methods used in designing the poll, first published in 1987.
Guterbock says the Best Places report, which is "built on an insufficient number of cases" and "contains inconsistent sampling," could potentially affect relocation decisions of thousands of households as well as economically impact the cities that are favored or avoided.
Money magazine groups the data obtained about the cities into nine categories: economy, health, crime, housing, education, weather, transit, leisure, and arts, which are then weighted according to readers' preferences. However, Guterbock says that "the data are combined by a formula that gives very high weight to economic indicators." Nowhere is it acknowledged that the rankings are based primarily on the short-term experience and outlook for each city, he says.
"The millions who read in their local papers that one town or another has just ranked high on the list of Best Places are being mislead if they think this is based on a global quality-of-life rating," says Guterbock.
He outlines suggestions for improving Money's Best Places poll, including exercising better control over sampling methods and adjusting the relative weights given to the survey factors.
"If Money magazine could produce reliable, stable, accurate, and honestly described economic ratings of cities, it would provide a useful and marektable service to its readers and to the public at large," says Guterbock. "In its present form, however, the Best Places 'poll' misleads the public and does and unfortunate disservice to the credibility of survey research."