How pressing is the need for school reform? In the late 1970s and throughout the 1980s, school critics issued a steady stream of reports claiming that the schools were in decline and recommending sweeping reform.[1] Then, as the 1990s began, a group of researchers--dubbed the "revisionists"--
systematically challenged the conventional wisdom, arguing that the schools were not declining at all and were even at historic highs on some indicators.[2]
     The past few years have seen a softening in the critics' position. Where once the conservative school critics trumpeted a general educational decline, many now acknowledge that educational trends have been level for over a decade and talk as much about preparing students for the 21st century as about recapturing an ancient and lost excellence.[3] The revisionists deserve credit for this shift in position. We should all have serious doubts about a so-called Golden Age of education and be leery of looking back wistfully for solutions. The revisionists' call for a renewed focus on the poor and on the neglected schools in our cities should be heeded.
     There is a danger, however, that the revisionists' message will be misconstrued as a call to abandon general reform. I am particularly concerned that as we come to recognize that the schools are not collapsing, we will overlook the deep, long-standing academic problems that plague our schools and which cry out for attention.

Achievement Trends

     Our best measure of recent achievement trends comes from the National Assessment of Educational Progress (NAEP). Every couple of years, NAEP tests large, nationally representative samples of students in a variety of subjects and repeats items to establish trends.
     I begin with literacy because it undergirds academic performance and is a perennial concern of educators. Here, a picture is worth a thousand words (see graph). Like reading, NAEP writing performance has been basically stable for over two decades.[4]

NAEP Reading Performance, 1971-1990

    
     Was literacy better in the past? In the mid- 1970s, Roger Farr and his colleagues reviewed a century of then-and-now reading studies and found that each generation of students had been better educated than the last. They concluded that, "anyone who says that he knows that literacy is decreasing is. . . at best unscholarly and at worst dishonest."[5] In the late-1980s, Carl Kaestle and I updated and expanded their work-examining basic, academic, functional, and job literacy-and found that, except for a minor decline during the 1970s, reading performance had been remarkably stable throughout the 20th century.[6]
     Other indicators also point to stable or even improving performance.[7] NAEP math trends have been as flat as reading's. High school completion rates have hovered in the 82-86% range for two decades (see chart). Enrollments in college preparatory math and science courses rose during the 1980s. Dale Whittington recently analyzed trends in historical knowledge and found that today's students are the match of yesterday's. There have even been U.S. improvements in some areas in the international assessments.[8]

     We've also lost little ground in producing high-scoring students. The proportion of 17-year-olds reaching the highest proficiency level (350) on the NAEP tests, for example, has been stable for several decades. On the SAT, there was a minor drop in top verbal scorers in the early 1970s (from 11% to 8%), but the graph has been flat since. The proportion in math scoring above 600 has been rising and is now slightly above late 1970s levels.[9]
     The picture is not all rosy, however. A few NAEP areas have declined a bit in recent years. 17-year-olds' civics performance and 8th graders' reading and writing scores have slipped slightly.[10] The case for long-term stability is limited because then-and-now research has focused on literacy and history knowledge, but has not yet covered math, science, literature, or general knowledge. SAT scores remain at low levels. Finally, many historical gains have stalled. Median educational attainment, for example, rose dramatically between 1940 and 1980, from less than 9 to over 12 years of schooling, but has changed little since.[11] Recent improvements on many nationally normed standardized tests have also stalled and may have reflected teaching-to-the-test as much as genuine improvement.[12]
     In the face of major changes in school populations and in society during the past decades, our schools, as measured by NAEP data, have demonstrated an enormous resilience and teachers should be commended for it.[13]

SAT Decline: Part Mirage, Part Real

     So far, I've ignored the SAT decline, but for good reason. The SAT is a poor barometer of general school performance-it is based on a selective sample of college-bound students (mostly from the top half of their classes) and not the typical student. This group does not even represent college-bound high school seniors many of whom take other admissions exams.
     Although there has been a huge apparent decline in SAT scores-verbal scores, for example, reached an all-time low of 422 in 1991-Math SATs, at their trough around 1980, have since risen, but remain below early 1970s levels-the decline largely reflects an enormous change in test takers that resulted from the 1960s and 1970s expansion of educational opportunity.[14] The SAT was normed in 1941 on about 11,000 mostly white, male, middle-class students bound for Ivy League colleges. Today's SAT is taken by over a million students, over a quarter are minority, and far more students have lower SES and lower high-school class rank. Given these students' weaker preparation and the often poorer condition of minority schooling, is it any wonder that scores dropped greatly from the 1941 norm of 500?
     The SAT decline, however, is not entirely compositional. The tremendous rise in minority test-takers, for example, cannot explain the large decline in white students' SAT scores during the 1960s and 1970s. During one stretch, the pool of test takers did not expand, yet scores still declined. This suggests that, to some extent, there was a real decline in performance.[15]
     The most comprehensive analysis of the demographic changes-the College Board's special Advisory Panel study published in 1977-concluded that much of the 1960s decline, from 2/3rds to 3/4ths, but a smaller part of the 1970s decline, up to 30%, was due to demographic changes in test takers.[16] If one considers the additional effects of age (students were getting younger) and birth order (later-borns score more poorly), up to one- half of the 1970s decline may have been due to compositional changes.[17] The Advisory Panel attributed the remaining portion to an undetermined combination of school and societal factors.
     Since the late 1970s, the SAT decline has been marginal and affected only verbal scores, which dropped 6 points. Math scores rose 9.[18]
     So, there was something of a decline on the SAT back in the 1970s-but what declined? Was there a measurable weakening of academic performance or was there a weakening in test-taking skills, motivation, ability to do problems quickly, or ability to do verbal analogies?
     There are strong indications that high school achievement among the college bound did not decline. During the period of the greatest decline in SATs, 1967 to 1976, students improved their performance on six major ETS Achievement tests, including English Composition, the three high school sciences (biology, chemistry, and physics), and the two major foreign languages (French and Spanish). The students taking the achievement tests had lower SAT scores than their predecessors but, nevertheless, had higher achievement scores. The Advisory Panel speculated that the SAT was changing in its relevance to high school education.[19] In the past decade, there have been increases on all achievement tests.[20]
     We also know that the average high school student did not lose whatever skills are tested by the SAT. A short version of it, the PSAT, was given to nationally representative samples of high school juniors five times from 1955 to 1983. Although the trends were not uniform, scores in the early 1980s matched those of the 1960s.[21]

Questionable Relevance

     The most serious limitation of the SAT data, however, is their lack of relevance as a measure of school quality. Those who hold the schools responsible for the SAT decline ignore contradictory evidence. In a background study for the Advisory Panel, Gary Echternacht compared educational changes in high schools with stable SATs and those with declines worse than the nation's.[22] Principals in the two sets of schools reported similar increases in truancy and discipline problems. Their schools had expanded pass- fail grading and nontraditional offerings to the same extent. English curricula and enrollments in academic courses were similar. The decline in SAT performance was little affected by what the schools were doing.
     Such a lack of school effects should not be surprising. The Educational Testing Service-which designs the SAT-has warned us for years against using it to measure school quality. Remember that the SAT is an aptitude test designed to predict college performance. Although the math section includes algebra and geometry problems, the SAT ignores most of the high school curriculum. There is no material from such courses as French II, Chemistry, U.S. History, or English Literature. As Lynne Cheney aptly put it,

College freshmen throughout the nation reveal a striking ignorance of even the most elementary aspects of United States history, and know almost nothing about many important phases of this country's growth and development, a survey just completed by THE NEW YORK TIMES has shown.

     The Times surveyed 7,000 students in 36 colleges. They found that only 16% of the students could name two contributions by Thomas Jefferson. Only 20% could name two figures connected with trusts or two areas purchased by the U.S. and the nations that sold them to us. Only 13 percent knew that James Madison was president during the War of 1812. Keep in mind that these college freshmen were an elite-college attendance was then rare; only 6% of those in their late 20s even had college degrees.[55]
     Like today's critics, the Times could not resist exaggerating students' ignorance. The reporter wrote that "They could not identify such names as Abraham Lincoln," but 75% knew he was president during the Civil War and 69% knew he was assassinated. One can understand the reporter's temptation. Students' answers included such classic errors as "Lincoln emaciated the slaves" and that Congress had the power of "appeasement" (impeachment). Several hundred credited Theodore Roosevelt for creating New Deal agencies; many included Texas among the original 13 states. Asked which presidents were assassinated, many listed non-presidents who were not assassinated such as William Jennings Bryan and Henry Clay. A student from a Southern college could not resist editorializing. "Lincoln," he answered, "and it was a good thing, too!"
     The next year's follow-up test by the freshly created Committee on American History in Schools and Colleges produced equally dismal results. The typical high school senior answered correctly only 22 out of 65 questions.[56] Historical knowledge may not be declining, but it has been poor.

Geography

     Students' geography knowledge has been unfairly described as "feeble".[57] Reports of students' inability to find things on maps are legion, yet NAEP's tests have shown that juniors and seniors know many major world regions.

     - 92% located the Soviet Union
     - 87% located Canada
     - 84% identified Middle East countries from a list
     - 76% located West Germany
     - 72% identified petroleum exporting countries on a world map
     - 71% located Latin America on a world map[58]

     These items show how rapidly things change-there is no Soviet Union or West Germany today. What some students didn't know doesn't exist any more.
     The NAEP geography report claimed that "few demonstrated an understanding of physical geography that extended beyond the basics."[59] But only two such items were answered by a minority of the students! Most knew the cause of the seasons and that the Hawaiian Islands were formed by volcanism, and understood weathering and soil erosion. Sixty percent knew about faulting and sedimentary deposits.
     Still, there were serious problems. Many of these questions were easy and a substantial portion of students-from 15-40%- did not answer them correctly. Most students could not interpret a graph showing birth and death rates. Given our country's Vietnam experience, it is unsettling that 63% of our high school seniors could not locate Southeast Asia on a world map. 64% did not know Saudi Arabia's location, but this was before the Persian Gulf War. Half could not answer the following:[60]

     Additional bad news came from the 1988 Gallup international survey of those 18 and older.

- "From outline maps, the average American can identify only four of twelve European countries, less than three of eight South American countries, and less than six of ten U.S. states."
 
- "One American in seven (14%) cannot identify the United States on a map of the world; one fourth cannot find the Soviet Union or the Pacific Ocean."[61]

     The geography problem is not of recent vintage. Although a Gallup then-and-now study showed adults knew more in the 1940s than in the 1980s, it also revealed that performance was weak then.[62] Respondents could identify only half of the European countries and this was just after the war! Geographic knowledge was also poor on the 1943 New York Times' history test. Only 15%-29% could identify which bodies of water the following cities were on: Cincinnati, Cleveland, Memphis, Milwaukee, St. Louis, and Portland, Oregon.

General Knowledge

     Surveys by national polling organizations in the 1950s and 1960s suggest that general knowledge has been weak for decades. In the late 1950s, for example, high school graduates, ages 25-36, averaged only 34- 65% correct on items dealing with domestic public figures and events, foreign figures, history, humanities, geography, and science. College graduates did much better, 62-89%, but few adults were college grads.[63]
     The high school graduates showed many surprising gaps. In domestic matters, for example, only about a third knew their congressman; less than half knew which party held a congressional majority. In foreign affairs, only about a third knew Adenauer and only about half knew Nehru. In literature and art, only 8% knew Rubens, one out of five who composed the Messiah, and 37% who wrote A Midsummer-Night's Dream. In geography, only about half knew that Montana bordered Canada or that Mt. Everest was the highest mountain in the world. Under miscellaneous items, only 6% knew which planet is nearest the Sun, 13% the major language of Brazil, and 42% how many feet are in a mile.
     Several recent national studies showed that civics knowledge in the 1980s was at the same modest levels it had been going back to the 1940s.[64] In spite of great improvements in educational attainment and the expansion of media, the public still knows little about the structure of government and often does not know basic political and economic concepts, or the names, countries, forms of government, and U.S. policies involved in major political events and issues.

Literacy

     Although literacy has not declined, it, too, remains at low levels. NAEP's writing assessments have provided some of the clearest evidence because they use explicit criterion-referenced scoring which is easier to interpret than the standard NAEP scaling. Papers are judged using four categories-unsatisfactory, minimal, adequate, and elaborated-and it would be possible for all papers to be rated at the highest level.
     NAEP analysts concluded that, although performance had been stable, "Many students continued to perform at minimal levels on the NAEP writing assessment tasks, and relatively few performed at the adequate or better levels." Here are the 1990 percentages of adequate and elaborated responses by 11th graders.[65]

Informative Writing

Describe a summer job and their qualifications for it [67% and 1%]

Create a newspaper story based on notesabout a haunted house
[49% and 2%]

Read a description of frontier life and explain why today's food differs
[18% and 1%]

Persuasive Writing

In a letter to their senator, explain their position on reducing space program funding
[27% and 1%]

Defend choice of railroad track or warehouse as a recreation center
[20% and 1%]

Take stand on bike lane installation and refute opposing view
[19% and 1%]

     Most students could not handle these tasks. One must question, of course, how well a set writing task measures students' writing ability. As the NAEP analysts put it, students had to "produce first-draft writing on demand in a relatively short time under less than ideal conditions." They were given 15 or 30 minutes. The guidelines for assessing the writing, however, were designed to "reflect these constraints and do not require a finished performance."[66]
     NAEP is changing the assessment. Time is being expanded to 25 and 50 minutes. Portfolios are also being assessed. As NAEP's Deputy Director reported, this means that NAEP can now report on students' best writing and not just their test writing.[67]
     There has been some good news on the writing front. Students are doing well with grammar, punctuation, and spelling. Most errors are made by only a few students. The NAEP analysts' conclusion is worth repeating.
     .while focused instruction in the conventions of written language may be necessary for certain individuals or subpopulations of students, additional whole-class drill and practice is not likely to be useful to the majority of students.[68]
     Like the writing assessments, then-and-now reading research gives us limited comfort. Although Carl Kaestle and I found that literacy trends had been stable for most of the century, we determined that much of the population-from 20 to 30 percent-has difficulty coping with common reading tasks. The latest national study of adult literacy produced similar distressing results.[69] We concluded that, "Even if schools today are performing about as well as they have in the past, they have never excelled at teaching minorities and the poor or in teaching higher-order skills. And, if the increase in education is the only reason the population has kept up with the increasing literacy demands of our society, there is plenty of reason for concern."[70] As noted, median educational attainment has stopped rising and knowledge is often limited.
     Given the evidence from these different subject areas, I would estimate that 1/4 to 1/3 of our nation's high school students have severe academic troubles, as many as half have serious deficiencies in several areas, and perhaps 1/4 are doing well. These are not definitive figures-there are too many measurement problems and unresolved issues about students' motivation for that. It is also easy to be critical of 17-year-olds' lack of knowledge. We must be mindful that many of us acquired much of our knowledge on our own or after high school-reading biographies and popular histories, traveling, taking history and literature courses in college, and following the news. Still, the low levels are troubling. They are unimpressive results for 12 years of schooling.
     Reform need not be predicated on the supposed economic consequences of low achievement levels. I have argued elsewhere that the Nation at Risk report made too much of a high-skilled, hi-tech future economy as a rationale for reforming education.[71] A complex, democratic society, however, still needs a well-read and knowledgeable citizenry and these findings suggest that the schools are not accomplishing this-and perhaps never did.

Part of the Picture!

     U.S. schools may not be declining, but they are struggling. A strong case can be made for far-reaching reform. The achievement data, however, represent only part of the picture. Reformers worry about other issues as well. Is the school environment safe, supportive, and worth learning in? Are students treated with respect by teachers and each other? Are they challenged intellectually? Are they developing healthy self attitudes? Do they have a work ethic? How well are students being prepared for democracy? Do they have a sense of ethics, citizen activism, and social responsibility? These may seem like quaint notions, but reform should not become a mere quest for higher test scores.
     We have hard data on some of these factors. The national assessments have gathered extensive data on teaching methods and student attitudes. Although there are a few bright spots, such as the frequent use of demonstrations in science classes, the portrait that emerges is troubling. NAEP analysts found that math instruction
     .continues to be dominated by teacher explanations, chalkboard presentations, and reliance on textbooks and workbooks. More innovative forms of instruction-such as those involving small group activities, laboratory work, and special projects- remain disappointingly rare.[72]
     History and civics classes appear dominated by textbooks, tests, quizzes, and short-answer questions. It is unusual to find students working in groups or writing long papers. Writing instruction in the schools is also limited and is focused on mechanics. Only about a fourth of 8th graders report that their teachers spend more than an hour a week on writing.[73]
     Interest in science has not been sparked. In 1986, fewer than a fourth of 11th graders reported working on science-related hobbies or talking with friends about science. Only about a third reported going to a science museum or trying to fix something electrical or mechanical.[74]
     Students do little schoolwork. The data on homework and TV watching are revealing. In 1990, only about a third of our 17- year-olds reported spending more than an hour per day on homework, whereas half reported watching 3 or more hours of television per day! Reading has been shortchanged. In 1986, over half the 11th graders reported reading on their own less than once a week; about a fifth reported they never did![75]
     One cannot look over this information without a sense that our schools are not what they could be or should be. But such quantitative data are not our only source of knowledge. Over the past decade, a series of thought-provoking ethnographies and school profiles has portrayed a school system in crisis.[76] What we're seeing, particularly at the high school level, is that students are often disengaged, teachers' work is often factory-like, and intellectual life is often poor.
     Reformers have been busy. They know that the schools are not better than ever, but rather, more than ever, they need to be different than they are. Teachers and other educators who are intimately involved in the life of schools recognize there is a serious problem. There are major reform efforts affecting every major aspect of education-curriculum , pedagogy , school organization ,governance , and evaluation .
     Given how deeply rooted and long-standing the academic problems are, such efforts should be coordinated and focused on a comprehensive structural overhaul of our nation's school systems. Thus, while our best evidence suggests that we can set aside concern over a recent decline and need not be prisoners of a wistful nostalgia, we must recognize that fundamental school reform is still warranted.

Endnotes

1. Frank Armbruster, Our Children's Crippled Future (New York: Quadrangle Books, 1977); Lynne Cheney, American Memory (Washington, D.C.: U.S. Government Printing Office, n.d.); Paul Copperman, The Literacy Hoax (New York: William Morrow, 1978); E.D. Hirsch, Cultural Literacy (Boston: Houghton Mifflin, 1987); National Commission on Excellence in Education, A Nation at Risk (Washington, D.C.: U.S. Government Printing Office, 1983); and Diane Ravitch, "The Uses and Misuses of Tests," College Board Review, Winter 1983-84, pp. 22-26.

2. Gerald Bracey, "Why Can't They Be Like We Were?, Phi Delta Kappan, October 1991, pp. 105-117; Robert Huelskamp, "Perspectives on Education in America," Phi Delta Kappan, May 1993, pp. 718-720; Iris Rotberg, "I Never Promised You First Place," Phi Delta Kappan, December 1990, pp. 296-303; Robert Rothman, "Revisionists Take Aim at Gloomy View of Schools, " Education Week, November 13, 1991, p. 1+; and Michael Wartell and Robert Huelskamp, Sandia National Laboratories, House Testimony, Hearings on the State of Education (Washington, D.C.: U.S. Government Printing Office, 1991), Serial No. 102-28, pp. 175-205.

3. Chester Finn, We Must Take Charge (New York: The Free Press, 1991), p. 12; and Gerald Bracey, "The Second Bracey Report on the Condition of Public Education," Phi Delta Kappan, October 1992, pp. 104-117.

4. Arthur Applebee et al, Writing Report Card (Princeton, N.J., NAEP, 1990), p. 6; National Center for Education Statistics (NCES), Condition of Education 1992 (Washington, D.C.: U.S. Department of Education, 1992), p. 42.

5. Roger Farr, J. Tuinman, and M. Rowls, Reading Achievement in the United States: Then and Now (Bloomington, IN: Indiana University, 1974), p. 140. ED 109 595.

6. Lawrence C. Stedman and Carl F. Kaestle, "Literacy and Reading Performance in the United States, from 1880 to the Present," Reading Research Quarterly, Winter 1987, pp. 8-46.

7. Ina V.S. Mullis et al, Trends in Academic Progress (Washington, D.C.: U.S. Government Printing Office, 1991), ED 338 720; NCES, The Condition of Education 1991 (Washington, D.C.: U.S. Department of Education, 1991), p. 55; NCES, Dropout Rates in the United States: 1991 (Washington, D.C.: U.S. Department of Education, 1992), p. 97; Dale Whittington, "What Have Our 17 Year Olds Known in the Past?," American Educational Research Journal, Winter 1992, pp. 759-780.

8. P. Keeves and A. Schleicher, "Changes in Science Achievement: 1970-1984," in J. P. Keeves (Eds.), The IEA Study of Science III: Changes in Science Education and Achievement: 1970 to 1984 (Oxford: Pergammon Press, 1992), pp. 263-290; and David F. Robitaille, "Achievement Comparisons Between the First and Second IEA Studies of Mathematics," Educational Studies in Mathematics, 21, 1990, pp. 395-414.

9. Mullis et al, Trends; p. 242; and NCES, Condition 1992, pp. 187, 196, 223.

10. Arthur Applebee et al, Writing Report Card, p. 6; Mullis et al, Trends, pp. 4, 159; NCES, 1992, p. 44.

11. The proportion graduating from college, however, has continued to rise. NCES, The Digest of Educational Statistics 1991 (Washington, D.C.: U.S. Department of Education, 1991), p. 17.
 
12. Robert Linn, M. Elizabeth Graue, and Nancy Sanders, "Comparing State and District Results to National Norms" Educational Measurement: Issues and Practice, Fall 1990, pp. 5-14; Robert Rothman, "Revisionists Take Aim É"; and Lawrence C. Stedman and Carl F. Kaestle, "The Test Score Decline Is Over: Now What?," Phi Delta Kappan, November 1985, pp. 204-210.

13. The increases in the number of immigrant students, non-English speakers, and classified students probably did not have much of a direct impact on average scores, because those with limited English proficiency and severe disabilities generally have been excluded from NAEP testing. See Ina V.S. Mullis, The NAEP Guide (Washington, D.C.: U.S. Department of Education, 1990), pp. 35-36; Mullis et al, Trends, p. 206. Nevertheless, such changes have made teaching more difficult in many communities and may have held down other students' scores.

14. National Center for Education Statistics, The Condition of Education 1992, p. 54.

15. Another piece of evidence. Recent test takers whose background was similar to the 1941 SAT norming group--white students with a college educated parent--still did more poorly than the original norm group on the verbal section--454 vs. 500. They scored comparably on the math. See Bracey, "Why Can't They Be Like We Were? Murray and Herrnstein argue that the white students' composition did not change, and thus most of the decline is real, but they had income data only on the bottom 10%, parental education data for the 1980s not the period of the decline, and no class rank data. Murray and Herrnstein, "What's Really Behind the SAT-score Decline?

16. Willard Wirtz et al, On Further Examination (New York: College Board, 1977).

 
17. Stedman and Kaestle, "The Test Score Decline Is Over," p. 205. About half of the recent fluctuations in verbal scores can be accounted for by demographic changes--see Rick Morgan, Cohort Differences Associated with Trends in SAT Score Averages (New York: College Board, 1991), ED 336 409. The effects might have been greater but the study lacked SES data.

18. Verbal scores rose in the early 1980s and then dropped in the late 1980s. The hub-bub over the average has obscured the fact that minority students generally improved their verbal and math scores during the 1980s. See NCES, 1992, p. 227. The SAT decline has been attributed to Simpson's Paradox (see Richard M. Jaeger, "Weak Measurement Serving Presumptive Policy," Phi Delta Kappan, October 1992, p. 120), but this ignores the decline in white students' verbal scores which fell 10 points between 1976 and 1991. Several minority group's verbal scores also declined in the late 1970s and again in the late 1980s. The paradox is irrelevant to math scores which rose during the 1980s.

19. Willard Wirtz et al, On Further Examination, pp. 22-3.

20. The number of achievement test takers has also risen. However, these students' SAT scores are also up. This could mean that they are simply a more talented group of test takers rather than that the college-bound now have higher achievement. Bracey, "Why Can't They Be É"; Admissions Testing Program, National College-Bound Seniors, 1983 (New York: College Board, 1983), p. 13; and idem, National College-Bound Seniors, 1992 (New York: College Board, 1992), p. 9.

21. Math PSAT scores declined in the 1950s and 1960s while verbal scores rose. The 1983 PSAT verbal score was about the same as the 1966 peak while the math score was about the 1960 level, though below the mid-1950s. Murray and Herrnstein, "What's Really Behind the SAT-score Decline?"

22. The study is limited, however, in that it was based on principal reports and not researchers' observations of the high schools. G. J. Echternacht, "A Comparative Study of Secondary Schools With Different Score Patterns," in appendix to Wirtz et al, On Further Examination.

23. Cheney, p. 8.

24. College Board, 10 SATs (New York: College Entrance Examination Board, 1988), pp. 110, 118.

25. The two verbal sections that count consist of 85 questions--25 antonyms, 20 analogies, 15 sentence completions, and 25 reading comprehension questions. The two math sections that count consist of 60 questions--20 quantitative comparisons and 40 dealing with arithmetic, basic algebra, and geometry. The experimental sections have either 40 or 45 verbal questions and either 25 or 35 math problems. College Board, pp. 12, 19; Adam Robinson and John Katzman, Cracking the SAT & PSAT: 1993 Edition (New York: Villard Books, 1992), pp. 31, 105-106.

26. The SAT's validity has received enormous criticism. David Owen, None of the Above: Behind the Myth of Scholastic Aptitude (Boston: Houghton Mifflin, 1985) and James Crouse, The Case Against the SAT (Chicago: University of Chicago Press, 1988).

27. Robinson and Katzman, p. xv.

28. Arthur Applebee, Judith Langer, and Ina V.S. Mullis, Crossroads in American Education (Princeton, N.J.: ETS, 1989), pp. 22-23; John Dossey et al, The Mathematics Report Card (Princeton, N.J.: ETS, 1988), pp. 41, 43; Ina V.S. Mullis, The Science Report Card (Princeton, N.J.: ETS, 1988), pp. 48, 51.

29. Russell Allen et al., The Geography Learning of High School Seniors (Princeton, N.J.: Educational Testing Service, 1990), p. 71; Lee Anderson et al., The Civics Report Card (Princeton, N.J.: Educational Testing Service, 1990), p. 29; David C. Hammack et al., The U.S. History Report Card (Princeton, N.J.: Educational Testing Service, 1990), p. 16; and Mullis et al., Trends, pp. 6, 7, 39, 77, 124.

30. The NAGB was set up by Congress in 1988 to oversee NAEP and create the new standards. Standards for science, geography, and U.S. history are due in 1994. The level-setting procedure has been based on item analysis. The math panels examined items from the 1990 math assessment to determine what proportion of students in a given grade, at each proficiency level, should be expected to solve each problem. Mary Lyn Bourque and Howard H. Garrison, The LEVELS of Mathematics Achievement (Washington, D.C.: NAGB, 1991), ED 342 685, pp. v, 10. Math objectives were established separately by a national steering committee that worked with state math supervisors, state panels, math educators, and NAEP's item development panel. Ina V. S. Mullis, "Developing the NAEP Content-Area Frameworks," Journal of Educational Measurement, Summer 1992, p. 113.

31. The NAEP scale actually ranges from 0 to 500, but virtually all students score between 150 and 350. Improvements have recently been made in the way scores are reported.

32. Emerson J. Elliott, A Preliminary Report of National Estimates from the National Assessment of Educational Progress, 1992 Mathematics Assessment (Princeton, N.J.: 1993), ED 351 397, p. 9. Overall, to reach the basic level, students had to answer correctly 45-48% of all the items on the test, to reach the proficient level, 68-73%, and, to reach the advanced level, 87-89%, depending on grade level. The proficient level reflects "solid academic performance." Bourque and Garrison, The LEVELS, pp. 3, 4, 6.

33. Ina V.S. Mullis, Jay R. Campbell, and Alan E. Farstrup, NAEP Reading Report Card (Princeotn, N.J.: Educational Testing Service, 1993), p. 76.

34. Given the limited findings based on the new NAGB levels, most of my discussion focuses on the basic 150-350 levels for which there are two decades worth of data.

35. John Carroll, "The NAEP Reading Proficiency Scale Is Not a Fiction: A Reply to McLean and Goldstein," Phi Delta Kappan, June 1988, pp. 761-764; Eleanor Chelimsky, NAGB Achievement Levels. Interim Report (Washington, D.C.: GAO, 1992), ED 342 281; Robert Forsyth, "Do NAEP Scales Yield Valid Criterion-Referenced Interpretations?," Educational Measurement: Issues and Practice, Fall 1991, pp. 3-9, 16; Jaeger, pp. 120-121; Leslie D. McLean and Harvey Goldstein, "The U.S. National Assessments In Reading: Reading Too Much Into the Findings," Phi Delta Kappan, January 1988, pp. 369-372.

36. The General Accounting Office report found that the "empirical validity" of the levels "has not been demonstrated" and that their "predictive accuracy is as yet unsupported." Chelimsky, NAGB Achievement Levels, p. 4. The effects are likely to be small. Test scores, for example, are only weakly correlated with measures of workers' job proficiency. Stedman and Kaestle, "Literacy and Reading Performance", p. 38.

37. The levels were set by pooling results across all age groups. McLean and Goldstein, p. 370; Mullis, The Science Report Card , p. 140; and Mullis, Trends, p. 218.

38. Recent reports by the General Accounting Office and the National Academy of Education have been quite critical of the new NAGB scaling procedure. Serious questions have been raised about the levels' reliability, validity, and utility. Chelimsky, NAGB Achievement Levels; Jaeger, pp. 120-121.

39. Chelimsky, NAGB Achievement Levels; p. 3.

40. Mullis et al, Trends, p. 76.

41. Dossey et al, The Mathematics Report Card, p. 43. .

42. Hammack et al, pp. 25, 26

43. Although the NAEP 150-350 levels were set up in the mid-1980s, results from the earlier years (percent correct on different types of problems) were placed on the same scale to provide two decades of trend data. There have been only slight fluctuations at the advanced level. Mullis et al, Trends, pp. 39, 77, 124. Elliott, A Preliminary Report , p. 9.

44. See, e.g., Thomas Carpenter et al, "Results of the Fourth NAEP Assessment of Mathematics," Arithmetic Teacher, December 1988, pp. 38-41; Carpenter et al, "Student Performance in Algebra: Results from the National Assessment," School Science and Mathematics, October 1982, pp. 514-531.

45. Carpenter et al, "Results . . .," pp. 40, 41.

46. Mullis et al, Trends, p. 306.

47. Only 22-33% could compute using data from a table, convert a decimal to a fraction, find the area of an irregular shape, estimate circumference, or solve multi-step problems. Keep in mind that this is an average for individual items. A smaller proportion would be expected to consistently succeed. Students did well (89-98%) on adding whole numbers and decimals, reading a line graph, interpreting data in a table, identifying parallel lines, and evaluating an algebraic expression. Mullis et al, Trends, pp. 302-309.

48. Mullis et al, The State of Mathematics Achievement (Washington, D.C.: U.S. Department of Education, 1991), p. 80.

49. Mullis et al, Trends, p. 218.

50. Bourque and Garrison, The LEVELS, pp. 28-32.

51. The reading assessment will have longer passages and be 40% open-ended. Council of Chief State School Officers, Reading Framework for the 1992 NAEP (Washington, D.C.: U.S. Government Printing Office, 1992), ED 341 958; and Mullis, The NAEP Guide , pp. 16, 18.

52. Brent Bridgeman, "A Comparison of Quantitative Questions in Open-Ended and Multiple-Choice Formats," Journal of Educational Measurement, Fall 1992, p. 269; Michael Martinez, "A Comparison of Multiple-Choice and Constructed Figural Response Items," Journal of Educational Measurement, Summer 1991, pp. 131-45; NAEP, The Third National Mathematics Assessment (Denver, Colorado: Education Commission of the States, 1983), Report No. 13-MA-01, p. 32.; and Vincent Rogers and Chris Stevenson, "How Do We Know What Kids Are Learning in School?," Educational Leadership, February 1988, pp. 68-75.

 
53. Ravitch and Finn, What Do Our 17-Year-Olds Know?, (New York: Harper & Row, 1987), p. 201. Students averaged 55% correct in history, but this was not unexpected. The test was apparently constructed using norm-referenced techniques which guaranteed that the average would only be around 50% correct! Whittington, p. 778. Looking at the same data, NAEP analysts reached a more moderate conclusion, finding that, "the majority of high-school juniors do have some basic information about U.S. history." Arthur Applebee, Judith Langer, and Ina V.S. Mullis, Literature & U.S. History (Princeton, New Jersey: ETS, 1987), p. 9, 10. For the 1988 history report, ETS analysts switched to proficiency levels and concluded: "In summary, the assessment results indicate that across the grades, most students have a limited grasp of U.S. history." Hammack et al, p.10. I believe this neatly illustrates how the NAEP proficiency level scaling helps create a more dismal portrait than may actually be the case.

54. Fewer than 1/3 of the students had studied all the eras covered by the test. The test was weighted towards more recent material--over 1/3 of it covered the Great Depression to the present. Performance was better the more eras students had covered. Ravitch and Finn, p. 34, 54, 62, 179.

55. Only 29% of 18-19 year olds were in school in 1940. NCES, Digest 1991, pp. 15, 17.

56. Matthew Downey and Linda Levstik, "Teaching and Learning History: The Research Base," Social Education, September 1988, pp. 336-342.

57. Ina Mullis, NAEP Deputy Director, quoted in Robert Rothman, "NAEP Geography Assessment Finds Knowledge Gaps," Education Week, February 14, 1990, p. 1.

58. Ravitch and Finn, pp. 51-53; Russell Allen et al, Geography Learning, p. 13.

59. Allen et al, p. 42.

60. Allen et al, p. 16.

61. The Gallup Poll Public Opinion 1988 (Wilmington, Delaware: Scholarly Resources, 1989), p. 162.

62. The Gallup Poll 1988, p. 162.

63. Herbert Hyman, Charles Wright, and John Reed, The Enduring Effects of Education (Chicago: The University of Chicago Press, 1975), p. 133 and Table 2.5. Only 11% of 25-29 year olds were college graduates in April 1960. NCES, Digest 1991, p. 17.

64. Michael X. Delli Carpini and Scott Keeter, "Stability and Change in the U.S. Public's Knowledge of Politics," Public Opinion Quarterly, Winter 1991, pp. 583-612; Samuel L. Popkin, The Reasoning Voter, (The University of Chicago Press: Chicago, 1991), pp. 34-35. This lack of civic literacy does not necessarily mean the electorate is ignorant or blind. Popkin argues that voters use many information shortcuts to reason about issues and candidates.

65. Applebee et al, Writing Report Card, p. 6, 7; Mullis et al, Trends, p. 362. These were primary trait scores which avoided the proficiency levels problems. Figures were rounded to the nearest percent. Writing is also evaluated using NAEP's traditional scaling.

66. Applebee et al, Writing Report Card, p. 7.

67. Mullis, The NAEP Guide, p. 29.

68. Nevertheless, half the papers frequently displayed awkward sentences (1/4 of the sentences). Applebee et al, Writing Report Card, p. 54, 58.

69. Irwin S. Kirsch et al., Adult Literacy in America (Washington , D.C.: National Center for Education Statistics, 1993).

70. Lawrence C. Stedman and Carl F. Kaestle, "Literacy and Reading Performance in the United States from 1880 to the Present," Chapter 3 in Kaestle et al, Literacy in the United States (New Haven: Yale University Press, 1991), p. 128.

71. Lawrence C. Stedman and Marshall Smith, "Recent Reform Proposals for American Education," Contemporary Education Review, Fall 1983, pp. 85-104.

72. John Dossey et al, The Mathematics Report Card, p. 10.

73. Ravitch and Finn, p. 188-194; Ina V.S. Mullis, Eugene Owen, and Gary Phillips, America's Challenge: Accelerating Academic Achievement (Princeton, N.J.: ETS, 1990), pp. 59, 60, 73.

74. Ina V.S. Mullis, The Science Report Card (Princeton, N.J.: ETS, 1988), pp. 112, 114.

75. Mullis, Trends, pp. 100, 137; Ravitch and Finn, p. 162.

76. Michelle Fine, Framing Dropouts (Albany, N.Y.: State University of New York Press, 1991); John Goodlad, A Place Called School (New York: McGraw-Hill Book Co., 1984); Susan Kameraad-Campbell, Teacher (Chicago: Contemporary Books, 1989); Jonathan Kozol, Savage Inequalities (New York: Crown Pub., 1991); Peter McLaren, Life in Schools (White Plains, N.Y.: Longman, 1989); David Owen, High School (New York: Viking, 1981); Theodore Sizer, Horace's Compromise (Boston: Houghton Mifflin, 1984); and Patrick Welch, Tales Out of School (New York: Viking, 1986).

The preceding text is Copyright (c) 1993, Lawrence Stedman. Phi Delta Kappan, 1993, 75(3), 215-225.
Reproduced by permission of the author. All rights reserved.