AN EVALUATION OF THE
CLEVELAND SCHOLARSHIP PROGRAM

AN EVALUATION OF THE

CLEVELAND SCHOLARSHIP PROGRAM

AN EVALUATION OF THE

CLEVELAND SCHOLARSHIP PROGRAM

     During the 1996-97 school year, the Cleveland Scholarship and Tutoring Program (CSTP) gave 1,996 students from low-income families scholarships to be used at any participating Cleveland private school, secular or religious. Established by the State of Ohio in March 1995, 6,244 applications were filed the following fall, and scholarships were awarded beginning in January 1996. CSTP's beginnings were then complicated by a court challenge to the program's constitutionality brought by the American Federation of Teachers as well as by other organizations and individuals. But two weeks before the beginning of the school year, the court rejected the suit brought by the plaintiffs, and 1,996 scholarship recipients attended fifty-five private schools in grades kindergarten through grade 3.

     This evaluation, issued by Harvard's Program on Education Policy and Governance (PEPG), reports the results of a survey of all parents who applied for a CSTP scholarship, including both parents of scholarship recipients and parents of non-recipients. It also reports test-score results for students attending two schools established in response to the creation of CSTP.

     In the spirited national debate over school choice, many arguments have been advanced concerning the kinds of families likely to participate in a choice program, their reasons for participation, the willingness of private schools to accept a broad range of students, parental satisfaction with the program, the extent of school mobility that occurs within choice programs, the reasons students leave private schools, and the impact of school choice on student test scores. Although the CSTP has been in place for only one year, it already provides an opportunity to learn more about each of these topics. The American Federation of Teachers issued a report on CSTP in July 1997 expressing a range of concerns about the program's viability. In the following evaluation, PEPG briefly reviews of some of the current debate on these topics, including the concerns voiced by the AFT, and then presents pertinent information on these topics from the CSTP experiment.

Origins of the Program

     In March 1995 the Ohio General Assembly appropriated monies expected to be sufficient to provide 1,500 scholarships worth as much as $2,250 each. Scholarship recipients were to be chosen by lottery. The scholarship covered up to 90 percent of a school's tuition, the balance coming from the child's family or another private source. The maximum amount provided was little more than a third the per pupil cost of Cleveland public schools, which in 1997 was reported to be $6,507. However, publicly funded transportation was provided to scholarship students, and some private schools also received other public funds.

     The legislation establishing CSTP allowed as many as 50 percent of the total number of scholarships to be used for students already in private schools. However, the Ohio Department of Education limited that percentage to just 25 percent of the total. To meet this objective, families seeking a scholarship were asked (in the fall of 1995) to indicate whether or not the applicant was currently attending a private school. Of the 6,244 applications received by CSTP, 29 percent or 1,780 came from students already attending a private school. In January CSTP held a lottery in which 375 scholarships were awarded to these applicants. Additional scholarships were awarded later, and, as of April 3, 1997, 21 percent (427 of 1,996) of the scholarships were awarded to students previously matriculated in a private school. The remaining 79 percent of the scholarships were granted to students who had previously been attending public school or who were beginning kindergarten.

     Ohio Department of Education rules also gave preference to students whose income fell below the poverty line. First, students from low-income families received larger scholarships. Students coming from families whose income was below 200 percent of the poverty line received 90 percent of their school's tuition, up to $2,250, while those students coming from families whose income was at or above 200 percent of the poverty line were eligible to receive $1,875 or 75 percent of their school's tuition, whichever was less.

     Secondly, low-income students were initially given a better chance of winning the lottery. Families were asked to include information on family income on their application form. Of the 6,244 applicants, 58 percent (3,602) reported incomes below the poverty line, and the initial lottery, held in January 1996 with considerable fanfare on local television, was limited to this group of applicants. However, many of those offered scholarships did not accept them, either because the CSTP office was unable to reach them, or because they did not come to the CSTP office to give proof of income eligibility or, if they did, were found to be ineligible, or for other reasons. We provide more information on this topic below.

     By the summer of 1996, CSTP also discovered that tuition at Cleveland private schools was less than originally estimated, making it possible to increase the number of scholarship recipients to nearly 2,000. To accommodate more applicants, the Ohio Department of Education then relaxed the rules, making eligible any family with income below 200 percent of the poverty line. But by the time this second, less visible lottery was held, the CSTP program was in the midst of a court challenge, making it quite unclear whether the program would actually begin in the fall of 1996. Also, CSTP reported that it was becoming increasingly difficult to locate applicants (due to changes of telephone and address). As a result, CSTP reports that the acceptance rate declined sharply, and it eventually offered scholarships to all of the low-income applicants that it was able to contact. (However, as we shall see, nearly half of the non-recipients of the scholarships contacted in our survey thought that they had never been awarded a scholarship.)

     In sum, those applicants whose family income was below the poverty line were given preference in two important ways: 1) they received larger scholarships; 2) they were much more likely to have received a scholarship when the more visible lottery was held in January.    

     CSTP planning and administration were seriously hampered by a law suit that tried to prevent implementation of the program. The court case dragged on into August 1996, and it was not until two weeks before the beginning of the school year that the court found no constitutional or other legal barrier to CSTP. In addition, private schools reported difficulties obtaining student records from the Cleveland public schools. And transportation arrangements were not established until well into the school year, making it necessary to transport many scholarship students by taxi. In short, the program began with enough uncertainty and confusion that parental satisfaction could not be taken for granted.

The Parent Survey

     Since CSTP was the first program in the country to offer state-funded scholarships to low-income, minority students that could be used at any participating private school, secular or religious, giving the program national significance, PEPG offered to conduct a telephone survey of participating families and to analyze test-score data made available to it. CSTP made available information that enabled PEPG to contact a sample of the 6,244 applicants, and the Ohio State Department of Education supplemented PEPG's own funds with a grant that allowed PEPG to double the size of this sample so that 32.4 percent of all applicants were surveyed. PEPG asked the Social Science Research Institute at Northern Illinois University to administer the PEPG-designed questionnaire, and, in order to save time and money, asked the Ohio State Department of Education to direct its funds to the data-collection agency, even though it was agreed that PEPG director, Paul E. Peterson, had the responsibility for submitting a report to the Department. This report fulfills that obligation. A copy of the agreement with the Ohio State Department of Education is appended.

     A random survey of families was conducted in the summer of 1997, after the completion of the initial year of enrollment in choice schools. Interviews were conducted with 1,014 scholarship recipients, and 1,006 applicants who did not enroll in the program.

     The survey data have certain limitations. CSTP was not set up as a randomized experiment. Although a lottery was initially used to determine which individuals would be offered a scholarship, CSTP eventually attempted to give scholarships to all low-income applicants. As a result, those receiving scholarships may have been the applicants who could be more easily reached by the CSTP office and who were willing to have their income verified. Despite this departure from a randomized experiment, the fact that CSTP was initially planned as a lottery seems to have produced similar groups of recipients and non-recipients. As we shall see, the recipients new to choice schools and non-recipients remaining in public schools usually had similar demographic characteristics. When demographic differences can be observed, the scholarship recipients are the disadvantaged group. Though the two groups may also have unobserved characteristics that differentiate them, if such differences are correlated with demographic differences, the comparisons are likely to be biased against finding positive programmatic effects.

     To obtain the 2,020 interviews required 3,437 telephone attempts. Only 5 percent of the attempts resulted in a refusal; other interviews were not completed because no contact could be made, usually because the respondent was no longer at the phone number provided PEPG by CSTP (see table 5.1).      hspace="8" WIDTH="1" HEIGHT="1">

     The completion rate for scholarship recipients (74.1%) was higher than for those who applied but did not receive a scholarship (48.6%). The survey thus better replicates the characteristics of all scholarship recipients than it does the characteristics of applicants who did not enroll in the program, who are designated as non-recipients. As can be seen in table 5.2, the income and ethnicity of the recipients new to choice schools who responded to the survey did not differ significantly from the income and ethnicity of all such recipients (though they came from slightly smaller families). However, as compared to the universe of non-recipients, those responding to the survey had higher family incomes, were more likely to be white, and come from smaller families. As a result, if these demographic characteristics are positively correlated with parental satisfaction, the positive effects on parental satisfaction reported below under-estimate CSTP's actual programmatic impact.

     The respondents are divided into the following groups, which are given the indicated column number in many of the tables that follow:

          1) Recipients who previously attended a public school or who had yet to matriculate in any school. We refer to this group as "recipients from public schools" or "new to choice schools."

          2) Non-recipients who attended public schools during the 1996-97 school year. Ninety-three percent of this group were not attending a private school at the time of application. We refer to this group as "non-recipients in public school."

          3) Recipients who previously attended a private school, termed "recipients from private schools."

          4) Non-recipients who nonetheless attended a private school in 1996-97. Twenty-eight percent of this group were attending a public school at the time of application (School Year 1995-96) but chose to go to a private school, despite the fact they did not receive a scholarship. We refer to this group as "non-recipients in private schools."

     Three types of comparisons among these groups are particularly pertinent:

     a) First and most importantly, a comparison between groups one and two answers the following question: Do recipients from public schools differ from non-recipients in public schools?

In all tables containing four columns, a single asterisk indicates whenever a statistically significant difference between groups one and two has been identified at the .05 confidence level. Double and triple asterisks indicate differences significant at the .01 and .001 levels, respectively. The text of this report concentrates largely, but not exclusively, on this comparison.

     b) Second, a comparison between groups one and three answers the following question: Do recipients from public schools differ from recipients from private schools? One or more "t" signs indicate whenever a statistically significant difference has been identified at the confidence levels mentioned above.

     c) Third, a comparison between groups three and four answers this question: Do recipients from private schools differ from non-recipients attending private schools? One or more "#" signs indicate whenever a statistically significant difference has been identified at the confidence levels mentioned above.

     Five sections of tables are attached. The first section provides information on applicants to all grade levels; the second section of tables reports results for those applicants enrolled in grades one through three; and the third presents results for those enrolled in kindergarten. Section four presents test score results. Section five reports sampling information.

     The second and third sections of tables enable us to detect distinctions between those enrolled in kindergarten as distinct from those enrolled in the other three grades. According to a report issued by the AFT in June 1997,

          It is impossible to determine how many [of those accepting scholarships to enter kindergarten] would have gone to private schools without a voucher--for example, for religious reasons or because they already had a sibling attending the same school. Moreover, it is also important to note that in FY 1996, the year before the voucher program began, financial cutbacks forced the Cleveland Public Schools to discontinue full day kindergarten. . . in all non-magnet elementary schools. . . . Since incoming kindergartners cannot be considered students already subjected to 'failing' public schools, it is reasonable to assume that the choice being exercised here was not so much to leave the Cleveland public schools--these children had not attended them--as it was to benefit from the day-care and educational advantages of full-day kindergarten.

     If this argument is correct, then the responses of parents of kindergartners may be quite different from those enrolled in grades one through three. For the most part, we were unable to detect important differences. Consequently, we shall concentrate on the results reported in the first section of tables that report on all applicants and on section four that reports test-score results. However, we shall note as we go along any instances when substantially different results were obtained when those enrolled in kindergarten were excluded from the analysis. To facilitate comparisons across sections, tables within each section are ordered in the same sequence; tables 2.1 and 3.1 correspond to table 1.1, tables 2.2 and 3.2 correspond to 1.2, etc.

Background Characteristics of Applicants

     Many of those critical of choice programs fear that disadvantaged families will not be able to participate equally in the program. In the words of a recent Twentieth Century Fund report, if school choice "becomes a strategy to . . . restrict lower-income students of color to an inferior education, then the divisions between rich and poor in this country, and the attendant social problems, will only increase." But a Heritage Foundation report counters that "school choice programs benefit minority inner-city students the most." The parental survey permits an evaluation of these claims.

     Survey results indicate that it is possible for choice programs to award scholarships to low-income recipients. As can be seen in Table 1.1, the average family income of scholarship recipients from public schools was less than that of non-recipients in public school. Similarly, the average family income of scholarship recipients from private schools was less than that of non-recipients attending private school.

     In other respects, recipients from public schools differed but little from non-recipients in public schools. They did not differ significantly in terms of their mother's education, their mother's employment, their family size, their family living arrangements, their residential mobility or their religious affiliation (table 1.1).

     No significant ethnic differences between the recipients from public schools and non-recipients remaining in public schools were observed for grades one through three (table 2.1). However, some differences in the ethnicity among the kindergartners in the two groups were detected. Two-thirds of the scholarships were given to kindergartners from minority background. But of those who applied, the percentage among kindergartners receiving scholarships was 32 percent white as compared to 16 percent white among those remaining in public schools (table 3.1).

     When kindergartners are excluded, those recipients from public schools are even more obviously a disadvantaged group than non-recipients in public schools. Their income is still lower, they are more likely to be living in single-parent households, and they are more likely to have lived for only one year at their current address (see table 2.1).

     As can be seen in table 1.2, scholarship recipients coming from public schools were somewhat less likely to have received special education than were non-recipients in public school (11% as compared to 18%). But they were also less likely to have been in classes for gifted students (8% compared to 18%). It appears as if some students who had special education needs, and/or who had been suspended for disciplinary reasons, had difficulty obtaining placement in private schools. Of the small number of respondents who said they could not obtain admission in a desired private school (79), 25 percent said their child had been receiving special services related to a disability or learning problem and 12 percent said their child had been suspended for disciplinary reasons (table 1.3). Clearly, school choice plans need to be designed and funded so as to encourage the formation of facilities that can adequately serve populations with special needs.

Reasons for Seeking a Choice School

     School choice advocates say they wish to empower parents by giving them a choice among schools. But some critics have suggested that families, especially low-income families, do not choose schools on the basis of school quality. The Carnegie Foundation for the Advancement of Teaching has claimed that "when parents do select another school, academic concerns often are not central to the decision." The Twentieth Century Fund report claims that low-income parents are not "natural 'consumers' of education. . . [Indeed,] few parents of any social class appear willing to acquire the information necessary to make active and informed educational choices." Similarly, the AFT report on Cleveland suggests that parents sought scholarships, not because of "'failing' public schools" but "for religious reasons or because they already had a sibling attending the same school."

     Not much support for such criticisms can be found in the responses given by the parents of scholarship recipients in Cleveland. Asked why they applied for a scholarship, 85 percent of parents new to choice schools said they wanted to "improve the academic quality" of their child's education (tables 1.4 and 1.5). Second in importance was the "greater safety" to be found at a choice school, a reason given by 79 percent of the recipients. "Location" was ranked third. Contrary to AFT's suggestion, "religion" was ranked fourth, said to be very important by just 37 percent. Finally "friends," were said to be "very important" by less than 20 percent of the scholarship recipients.

     Non-recipients who remained in public school ranked the reasons in the same order but did not give any of them the same degree of importance as recipients from public schools did. Apparently, obtaining access to a private school was more important to recipients than non-recipients.

Reasons for Non-Participation

     Those who did not receive a scholarship were asked their reasons for not participating in the program. According to CSTP, the office had strong efforts to give all applicants an opportunity to receive a scholarship, so non-recipients may have had substantive reasons for not accepting a scholarship. The recent evaluation of CSTP by the American Federation of Teachers explained non-participation this way:

     It is clear that the [CSTP] Office made repeated efforts to make vouchers available to low-income public school families. However, some families who had originally applied for a voucher never followed up, as evidenced by the fact that families representing 34 percent of public school students in the voucher lottery did not visit the Office and verify their income. More significantly, many families who did verify their income and thus wanted vouchers could not find an available seat in a private school, at least not in the private school of their "choice". . . . About half of public school students who wanted vouchers most likely could not find an open seat in the private school or schools of their "choice."

     What then were the reasons many people apparently offered a scholarship by CSTP turned it down? The most important reason, from the parents point of view, was inadequate communication between CSTP and the applicants. No less than 44 percent of the non-recipients remaining in public schools said they were never offered a scholarship (table 1.6). In other words, almost half the respondents said they had not been offered a scholarship, despite the fact that our survey team was able to contact them one year later, when establishing contact was still more difficult. We estimate that well over a majority of those our survey team could not contact were also not reached by CSTP.

     We do not report this finding as a criticism of the CSTP office, which reports that it made extensive efforts to contact applicants. Low-income families are highly mobile and often depend on friends and relatives for telephone and mail service. They can be extremely difficult to reach. Even when contacted, many families may not have clearly understood that receipt of a scholarship required that they verify their income. Moreover, Department of Education eligibility rules, which first limited eligibility to those with incomes below the poverty line, may have discouraged many of those above the poverty line from giving the program further consideration, despite the fact that eligibility requirements were subsequently relaxed. Families with incomes above 200 percent of the poverty line were not eligible until November 1997, two months after the school-year had begun. Then, too, it was not clear until two weeks before the school year began that CSTP would survive the litigation being waged in Ohio courts. Finally, CSTP was given by the State of Ohio only a small staff to run a large-scale, complex innovation. Any new program can be expected to have growing pains in its initial year.

     All these factors complicated communications between CSTP and potential recipients, making it less than surprising that half or more of the non-recipients thought that they had not been awarded a scholarship. More surprising is that nearly 2,000 applicants did in fact receive a scholarship and were placed in a choice school within a short period of time--under difficult and continuously changing circumstances. That accomplishment was realized only by the dedicated efforts of the CSTP staff.

     Many of CSTP's first-year problems have eased as CSTP has entered its second year. The courts ruled in July 1997 that the program may continue a second year. Records from public schools have been acquired; relationships between participating schools and CSTP staff have matured. The staff is a more experienced team. As a result, the CSTP office reports that the number of scholarship recipients in the fall of 1997-98 has increased by 50 percent to approximately 3,000 students.

     Non-recipients who said they had turned down a scholarship were asked their reasons for not accepting it. Three reasons were mentioned with roughly equal frequency: transportation, financial considerations, and the offer of an admission to a desired public school (table 1.6). Apparently, the initial difficulties in setting up travel arrangements may have affected parental decision-making. In mentioning financial considerations, parents may have been referring to the fact that they needed to supplement the scholarship with a tuition payment (that could be as much as 25 percent of the cost), or they may have been referring to eligibility requirements, which initially limited scholarships to those below the poverty line. The third major reason, said to be important by over a third of the non-recipients in public schools, was their success in gaining admission to a desired public school. Observers reported that the Cleveland public schools responded to the scholarship program by giving scholarship applicants access to one of the city's magnet schools or enrichment programs. If so, it suggests that CSTP increased choices of CSTP applicants within the Cleveland public schools.

     The fourth most important reason for non-participation, given by 21 percent of those remaining in public schools, was the inability to secure admission to their desired private school (table 1.6). Many private schools were already oversubscribed. Also, the choice program was being set up in a context seriously complicated by the court suit filed by the AFT, creating a great deal of uncertainty as to whether it would become operational, making it is entirely possible that many private schools might have been reluctant to accept a large number of scholarship recipients from public schools. If legal issues are resolved, these problems can be expected to attenuate with time.

Parental Satisfaction

     Many economists think that customer satisfaction is the best measure of school quality. According to this criterion, there is little doubt that Cleveland's choice schools out-performed the city's public schools. Recipients from public schools were much more satisfied with every single aspect of their school than non-recipients in public schools (see tables 1.7 and 1.8). Two-thirds of parents new to choice schools report being "very satisfied" with the "academic quality" of their school, as compared to less than 30 percent of public-school parents. Nearly 60 percent were "very satisfied" with school safety, as compared to just over a quarter of non-recipients in public-school. With respect to discipline, 55 percent of recipients from public school, but only 23 percent of non-recipients in public school, were very satisfied. The differences in satisfaction rates were equally large when parents were asked about the school's "private attention to the child," "parent involvement," "class size" and school "facility." The most extreme differences in satisfaction pertained to "teaching moral values": Seventy-one percent of the recipients from public schools were "very satisfied," but only 25 percent of the non-recipients in public schools were.

     It is also of interest to inquire whether scholarship families coming from public schools were as satisfied as those who had already been enrolled in private schools. The AFT, in its report on the program, suggests that scholarship recipients from private schools were given important advantages. In the words of the report:

          Voucher students who had previously been enrolled in private schools held a "monopoly" on placements in the established private schools. In contrast, almost half of the voucher students who moved from public to private schools were enrolled in four schools with little or no educational and financial track record.

     There is some evidence from the survey in support of the AFT suggestion that scholarship recipients from public schools had a less satisfying educational experience than those from private schools. However, the differences between the two groups are, in most cases, modest. For example, 67 percent of recipients from private schools (table 8, column 3) say they were "very satisfied" with the academic quality of the school, while 63 percent of those from public schools (table 8, column 1) gave the same response. For school discipline, the percentages were 62 percent and 55 percent, respectively. The biggest difference was with respect to school safety, 69 percent as compared to 59 percent. Overall, choice parents, whether new to choice schools or whether they had previously been attending them, both expressed much higher levels of satisfaction with their school than did families with children still in public school (table 8, column 2).

     In sum, when one considers the fact that CSTP was not given the green light by the courts until mid-August 1996, it is a considerable accomplishment for CSTP to have provided "very" satisfying educational experiences for high percentages of its scholarship families, whether they came from public or private schools.

School Mobility Rates

     Most educators think that, all things being equal, it is better that students stay in the same school, especially during a single school year. Most of the time, education works better when it is not subject to disruption. An evaluation of the Milwaukee choice program has claimed that "attrition" from the program was its "most troubling aspect." But writer Daniel McGroarty has argued that mobility among schools was less in Milwaukee's school choice program than the mobility among the city's public schools. Moreover, high rates of mobility, he says, are to be expected given the high residential mobility rate that occurs among low-income, inner-city families. According to the U. S. census, the annual residential mobility rate among central-city, female-headed households with children between the ages of six and seventeen is 30 percent for African Americans and 35 percent for Latinos.

     In Cleveland 25 percent of recipients from public schools and 19 percent of non-recipients in public schools reported having moved in the past two years. (This was the "stable" portion of the applicant pool; it is almost certain that the applicants not reached in our survey were considerably more mobile.)

     Not every change in residence dictates a change in school attendance. But in Milwaukee's public elementary schools, nearly 20 percent leave even before the end of the school year in June. Come the following fall, nearly 40 percent of the students were no longer in attendance at the same public elementary school they were at one year previously.

Mobility within the School Year

     When these facts are taken into account, the Cleveland choice schools would seem to have done well in retaining their students. Only 7 percent of all scholarship families reported not attending the same school for the entire year (table 1.9). Among recipients from public schools the percentage was 10 percent. This mobility rate is approximately the same as the within-the-school-year mobility rate in the Milwaukee elementary school choice program and about half the mobility rate in Milwaukee public elementary schools.

     The most important reason recipients new to choice schools gave for changing schools mid-year, given by 3.3 percent, was admission to their preferred private school. Very likely, many of these changes were due to the fact that full establishment of the CSTP program was delayed by the legal suit, so that some recipients did not receive their scholarships until after the school year had begun. If so, this cause of school mobility should attenuate over time. Another one percent of the students changed schools because they had been admitted to a preferred public school; perhaps Cleveland public schools had given a magnet school opportunity to these families. Another 0.8 percent moved during the course of the year. And 0.8 percent moved for transportation reasons, perhaps a sign that the initial transport problems of the CSTP program posed difficulties for some parents.

     All of these reasons for changing schools are understandable and some can be expected to attenuate as the program becomes more established. But a total of nearly 2 percent of the parents said they changed schools because of the "quality " the school, administrative problems, school closure or change, and one-half of one percent of the families said expulsion, disability, or behavior problems had led to a school change, signs that CSTP was not working well for some families. The issues raised by these findings are discussed below.

Mobility between School Years

     Parents were also asked whether they planned on sending their child to the same school next year. A positive response was given by 81 percent of scholarships recipients from public schools, and 88 percent of the recipients who already had been attending private school (table 1.10). If actual choices are consistent with these plans, the mobility rate in Cleveland from one year to the next is approximately the same as in Milwaukee's state-funded elementary choice program, which was about one-half the mobility rate in Milwaukee public schools.

     Choice critics have suggested that private schools may routinely expel or not readmit students for a second year, if they are not keeping pace with their peers. Defenders of private schools say they use their authority to deny readmission sparingly. To ascertain whether low-income parents were encountering difficulties in obtaining readmission for a second year, those changing schools were asked the reason for the change. Less than one-half of one percent of recipients from public schools said their child could not be readmitted to the private school in which they had been enrolled. In other words, while refusal to be readmitted is not unknown, neither has it been practiced to any significant degree.

     Parents instead gave a wide range of other reasons for planning to move their child to another school in the fall of 1997. Six percent gave quite practical reasons, such as the family's moving from the area or the child's change in grade level (necessitating a school change). Another one and one-half percent found another private school they preferred, and a half percent found a preferable public school. Either transportation difficulties or financial costs posed an obstacle for another 2.4 percent. But 6.5 percent of all recipients from public schools were leaving because they were not satisfied with the quality of the school or were disappointed in the way in which the CSTP program operated. For a small but still important fraction of scholarship recipients, CSTP, at least in its first year, was not a success. Choice critics may see this as a sign of program failure, because not all families' expectations were fulfilled. However, school choice supporters may interpret this as evidence that choice allows a parent to make a move when things do not seem to be working out.

Test Scores

     Standardized test scores from two private schools with large enrollments of scholarship students, Hope Academy and Hope Ohio City, were made available to PEPG during the summer of 1997. An analysis of scores from these two schools is of particular interest for several reasons:

     1. The Hope schools were the only schools formed in response to the adoption of the Cleveland scholarship program. They thus provide information on schools that develop in response to the introduction of a parental choice program.

     2. The Hope schools announced they would accept all students who applied for admission. Many of the poorest and most educationally disadvantaged students went to the Hope Schools, making an examination of test scores from those schools a hard test case of the program as a whole. If gains are achieved here, they are probably being achieved under better circumstances in other choice schools.

     3. Enrollment at the Hope schools constitutes approximately 15 percent of the total enrollment in the Cleveland scholarship program and approximately 25 percent of those students who had previously attended public schools.

     4. The Hope schools were the only schools to test students in both fall and spring, thereby allowing for an assessment of whether students made test score gains.

     The AFT has expressed concern the Hope schools, with its large proportion of scholarship students, were "voucher-dependent" and had little or no "educational track record." Because a large percentage of students who were drawn from public schools went to the Hope schools, the AFT suggested that these scholarship students were being channeled into inferior institutions. An analysis of test scores from the Hope Schools helps address these concerns.

Scores from the California Achievement Test (CAT) were examined for all students who had scores from both the fall and spring of the 1996-97 school year. The Hope Schools' staff reported that they tested all students in attendance when the tests were administered, including those students identified as having special needs. Proctors from John Carroll University were present during the Spring testing to ensure the integrity of the testing process.

     The scores of Hope-school students show moderate gains in reading and large gains in math. After one year, students in kindergarten through third grade scored, on average, 5.4 percentile points higher on the reading test and 15.0 percentile points higher on the math concepts test (Table 4.1). Reading scores of students in first through third grade increased by 5.4 percentile points, math concepts scores by 12.8 percentile points, and total math scores by 11.5 points (Table 4.2). (Only the math concepts test was administered to kindergarten students.) Improvements were experienced by students in all grades. These results were statistically significant at the .05 level.

     The language test score, however, shows a decline of 4.8 percentile points. This decline was produced by a 19.0 point drop among first graders. Second graders improved by 2.9 points and third graders gained 12.9 points on the language test (Table 4.3). Kindergartners did not take this component of the test. The language test is not normally administered by the Cleveland Public Schools.

     The Hope Schools' test results can be put in perspective by comparing these results to those of similar students elsewhere. The Hope School students are distinctive in that they are poor, mostly of minority backgrounds, and applicants to a state-funded scholarship program. Test scores gathered in the spring of the year are available from a similar group of low income, minority, scholarship applicants who entered the first four years of the Milwaukee choice experiment (1990-93) at a time when they were still in public schools. These students had spring standardized scores that were lower than those of the Hope-school students. The spring Hope-school math test scores, on average, were at the 40.8 percentile, as compared to the 34.9 percentile in Milwaukee. On math concepts, Hope school students scored 46.9, as compared to 31.0 in Milwaukee. On the reading test, Hope school students scored at the 34.8 percentile, as compared to 33.5 in Milwaukee (see Table 4.4).

     The gains in percentile rankings achieved by Hope-school students need to be contrasted against the 1 to 2 point decline that is typical of inner-city students. According to data provided PEPG by the office overseeing desegregation in Cleveland, Cleveland public school reading scores declined, on average, by 1 to 2 percentile points between both the first and second grades and the second and third grades in the years 1994-95 to 1995-96. PEPG and other researchers found a similar pattern in the Milwaukee choice experiment. The decline in percentile rankings as inner-city students get older describes a situation in which students learn at a slower rate than the national averages and therefore continue to fall further behind in percentile rankings. Gains in percentile rankings, as were observed at the Hope schools, suggest that students are learning at a faster rate, allowing them to close the gap with others nationwide.

     More definitive conclusions about the effects of the scholarship program on academic achievement depend upon the collection of additional data. It is possible that some of the gains achieved by students at the Hope Schools will fade over the summer in the absence of enriching activities. It is also possible that the results at the Hope Schools are not representative of the program as a whole. But initial data from two of the schools deemed most problematic by choice critics suggest that CSTP has helped improve student test scores.

Conclusions

     The data upon which this evaluation is based are necessarily limited. CSTP has been in operation for only one year, a period not long enough to evaluate fully an educational program. Test-score results need to be monitored over several years before definitive results can be obtained. Also, CSTP was not set up as a randomized experiment, enabling investigators to compare participants with a control group of essentially similar parents and students. The results from the parent survey reported above thus compare groups that may differ in respects that cannot be detected.

     Despite these limitations, the quality of the data is sufficient to draw some preliminary conclusions. The survey of parents was able to interview over 70 percent of scholarship recipients the survey team attempted to reach, and the number of recipients interviewed (1,014) was more than half the universe of recipients (1,996). The demographic characteristics of the sample of recipients are not significantly different from those of the universe from which they were drawn. The survey team was less successful in reaching non-recipients, contacting 49 percent. The non-recipients surveyed were from more advantaged backgrounds than those of the universe from which they were drawn; however, this limitation to the data set simply biases the findings against the parental satisfaction results that were obtained. It is likely that even stronger results would have appeared had the response rate been higher.

     Though the comparison groups were not created by a randomized experiment, neither did their background characteristics differ in important respects. Whenever demographic differences do appear, it is the scholarship recipients from public schools who are usually the more disadvantaged group. Kindergartners are an exception to this generalization, but similar findings appear, whether or not kindergartners are included in the analysis.

     The parent survey indicates that CSTP has succeeded in providing educational opportunities to low-income families that have won a strong endorsement from those participating in the program. A majority or near majority of parents are "very satisfied" with nearly every aspect of the schools they attend, and levels of satisfaction with choice schools are much higher than the levels of satisfaction with Cleveland public schools.

     Test scores in math and reading have risen in the two schools newly established in response to CSTP, which choice critics deemed among the most problematic of those participating in the program.

     Parents listed academic quality as the most important reason for their participation in CSTP, suggesting that educational objectives are paramount in their choice of school. However, a fifth of the non-recipients said a reason they did not participate was their inability to find a desired private school. And families whose children had special education needs found it more difficult to obtain a desired private-school placement.

     School mobility rates were lower than in central-city public schools. Only a tiny fraction, less than one-half of one percent, of the parents new to choice schools reported that their child had been expelled from their private school or refused admission for a second year.

     In sum, though CSTP encountered some difficulties establishing itself in its initial year, in good part because a legal suit introduced a good deal of administrative uncertainty, both test score and parental survey data provide strong justification for the legislative decision to continue and expand CSTP for another year. However, special funding arrangements and further programming would seem necessary, if disabled and other special needs students are to participate fully in a school-choice program.