By William J. Mathis
Policy wonks, not getting the results they hoped for (and being a touch lazy) invented a new research method called the “intraocular traumatic test.” What that means is, look at the data and if the result hits you between the eyes, it’s significant. Naturally, this kind of eyeballing lends itself to people seeing what they want to see. Talking heads and blog commentators are the most virulent users of the method. Educational policymakers and call-in radio shows are also malignant carriers of the malady.
The latest outbreak is in the release of the National Assessment of Educational Progress (NAEP) scores. Over the past 30 years, the invariable reaction is Chicken Little crying, “The scores are falling! The schools are failing!” Innocent bystanders spot the use of the trauma test by the intense volume, lack of proofs, vaporous claims; all garnished with a plug for a favored solution.
How big are the actual losses? In the instant case, we need to take a close look. Since 1990, NAEP scores show steady increases until the recession in 2008. Then the national progress slows down. The biggest national loss was three points. That does not sound like much – because it isn’t much— a meager three points on a 500-point scale. This is less than 1% over two years. If we go back to 1990, the results show continuous gains across the board. Once the test error is considered, the “drop” is less than 1% over two years.
The threshold issue is poverty. In a recent Stanford study by prominent researcher Sean Reardon, he examined 350 million test scores from every public school in the nation. He found that poverty is the key predictor of test scores.The analysis included social as well as school factors.
If Maslow’s fundamentals of food, shelter, and nurturing are not available, the chances of the child learning quadratic equations is remote. Only then can we effectively address the tangled web of needs. Beyond addressing poverty there is no single in-school or out-of-school factor that explains the scores Many just blame “the schools” and offer tired and vaporous generalities. As Brookings points out, some states are still not back to pre-recession financial levels. The effects of “the lost decade” still reverberate in unique and unpredictable ways.
Nevertheless, some potentially illuminating considerations are worth exploring.
Recent commentators look at one test given to only a sample and conclude that the schools are failing. The blind eye of the inter-ocular trauma test simply doesn’t have the necessary breadth of vision. It is like recruiting baseball players on their batting average. Whether they are good fielders, base-runners, or team mates is not considered. If we are to measure education accurately, we need long-term data on children as well as qualified visiting teams.
Over various administrations, policy makers have taken steps in the name of reform that are a blend of Keystone Kops armed with food processsors:
We have only small gains on the threshold issue of concentrated poverty. Some schools need better targeted resources.
An unintended consequence of Act 46 was a political backlash. The distracting cacophony was like banging a garbage can in a concert.
Computers: Below the national average, only 80.9% of Vermont households have internet. Expecting these students to perform well on the computerized NAEP test was unrealistic.
Proficiencies: Redefining all content as “proficiencies” was done without the necessary support capacity. In a statewide survey, 50% of the teachers feel uncomfortable with proficiencies.
Too many curriculum changes. Saying local districts decide curriculum is disingenuous. We have adopted the common core and numerous other curriculums over the years. There has not been consistency, stability or support.(Grade expectations, CCSS, NGSS, NCAS)
Too many test changes –The relation between NAEP and SBAC accounts for only 30% of the variance in reading and 24% in math. The tests measure different things and send educators in conflicting directions. SBAC NSRE, NAEP, NECAP, Science, SBAC, NSRE.
Well-intentioned programs often have a regressive financial effect. Pre-school education has an inequitable funding system, dual enrollments favor more affluent children, and tracking provides a different quality of education across the student population.
The greatest intraocular traumatic shocks have been self-inflicted. The schools are not failing as much as we have failed to provide robust programs for our neediest. They are often undercapitalized, lack coherence, steadiness and purpose.
Yet the loss of three-fifths of one point on NAEP must be viewed in context.
The intraocular traumatic test does not indicate a failure of educators. Instead it demonstrates a hard-won success with limited support, overcoming hard challenges and great sacrifices. If we are to improve, it will not be done with castigations, it will be done with co-operative and focused work based upon an appreciation of all that we have done.
William J. Mathis is the managing director of the National Education Policy Canter. He lives in Goshen. The views expressed are strictly those of the author.
