From Findings to Implications

One of the most significant reports to be published this year is from the Center for Research on Education Outcomes (CREDO). Their National Charter School Study carefully examined achievement gains by students in charter schools to those in traditional non-charter public schools.

This report is so significant, in large part, because of the breadth of their sample. They included roughly 70% of charter schools students in their study, students spread across many states with different charter school laws. Furthermore, they look at data in the level of individual students, comparing each charter student to another student from their same feeder school -- one matched on grade level, gender, race/ethnicity, free or reduced price lunch status, English language learner status (ELL), special education status and prior test score on state achievement test (p. 16).

This is good stuff. Yes, it is limited, but all researcher face practical limits. In this case, they could not address home or family factors (e.g. number of adults in the home, parents educational level, etc.), because legally they could only get access to certain data. From what I can tell, they did a great job.

My concern stems from how people read their findings and extrapolate too far to implications, something that happens quite often.

In this case, the reachers found that "Two subgroups fare better in charters than in the traditional system: students in poverty and ELL students." I do not question their methodology or their quantitive analysis. However, I think that they are thoughtfully presenting the limits of their sampling strategy -- limits imposed by the nature of the issue they are addressing.

One critique of charter school studies is that they do not account for self-selection bias. The authors of this study acknowledge the issue, and directly address how they attempted to deal with it. But I think that the problem is a bit more nefarious than they let on. Self-selection is only a problem if the self-selectors might be different than the non-selectors in other relevant ways. When it comes to charter schools, this might well be the case. I believe that charter school self-selecting families have a greater commitment to education/schooling than the general population, and a greater sense of agency (i.e. "something that I can do can make a difference"). Transmitting these sorts of values to their children is one of the many ways that families help children to succeed in school.

I think that this difference is especially pronounced among families in poverty and families of English language learner students. When these families step up and find what they believe is a better school for their children -- regardless of whether or not they are making a good decision -- they are modeling important behavior for their kids. This is not to say that demographically similar families who do not choose charter schools necessarily lack these traits. Not at all. Rather, I believe that they are significantly more common among those self-selectors.

I truly believe this finding (i.e. that ELL students and students from poverty do better in charter schools) is merely the self-selection issue at work. Unfortunately, the report's authors fail to address the issue of self-selection in their own implications section (pp. 49-51).

Before this report game out, I believed that charter schools would -- in the long run -- perform just about the same as traditional non-charter public schools. However, various findings in this report have convinced me otherwise. I do not know if leaving out the remaining effects of self-selection was merely an oversight on their part, or some kind of misguided attempt to boost their own credibility by providing supports for charter proponents' claims. But I am concerned that those proponents will latch on these particular findings without acknowledging the real dynamics at play.

And so, I wish that CREDO had presented their implications a little more thoughtfully.