What is "The Gold Standard"?

Did you hear about the big report that came out this week? You know, the one that "shows" that NYC charter schools are better than traditional non-charter public schools? It has gotten a ton of attention, probably because it uses "'the gold standard' method[ology]." The report is not subtle about this. It is right there in the very first sentence of the executive summary, "The distinctive feature of this study is that charter schools' effects on achievement are estimated by the best available, "gold standard" method: lotteries." It even uses the term "gold standard" four more times throughout the report.

Everyone wants to follow The Gold Standard -- or at least be able to say that they do. Of course! I mean, who wouldn't? But I do not think that we actually have a gold standard in education research. In fact, I am quite sure that we do not, and appropriating biomedical research's gold standard does not make it appropriate for us.

However, if we are going to borrow their standard, can we not at least get it right?

The biomedical standard uses double-blind experimental studies with random assignment. That means that some research participants get the experimental treatment and some get a placebo, and both are assigned randomly. It also means that neither the researchers nor the participants know who is getting which treatment. After all, expectations are important, and the mind can set us up for all kinds of things.

*****************

One of the latest ideas about The Gold Standard in educational research concerns charter schools.

We all want to know whether charter schools are better than traditional non-charter public schools. On one level, we certainly do want to know about individual schools. But on the policy level, we want to know about the average charter school, because we want to figure out if "charterness" helps a school be better. If it does, then we want more charter schools. If it does not, then we want fewer or none. And if we cannot be sure, we want to keep checking.

Let me say this quite clearly: Some charter schools are better than most non-charter public schools, and some are worse. And some non-charter public schools are better than most charters, and some are worse.

The Gold Standard crowd have a favorite method for comparing charter schools to non-charter public schools, one of which they are quite proud, but one that is so full of problems that I am shocked that they keep using it.

They rightly want to control for self-selection bias among charter school students. We know that children and families that apply to charter schools are different from those who do not, even if we do not know what all those differences are. This seems like the perfect time to do a randomized assignment, because that is the best method to make sure that these differences cancel out between groups. Luckily, we have some randomized assignments. Oversubscribed charter schools are virtually always supposed to accept students using a random lottery. This allows researchers to compare the outcomes of those were were randomly accepted, and those who were not.

Sounds good, right?

Well, it does sound good. But serious issues remain. Some are more obvious than others, and some are correctable by those interested in getting the correct answer, rather than the one that fits their pre-ordained conclusions.

Issue #1: No Placebo

Biomedical research does not just include randomization of treatment. It also is at least single blind. If some patients know that they are getting the new treatment, they might react differently. The mind is a powerful thing. They might be more diligent. Perhaps do their rehab exercises more often. Maybe pay more attention to diet. Who knows? And those who know that they are getting the new treatment might not lose hope.

If a student and/or his family do not get into the school of his/her/their choice, how might they react? I know from my own experience teaching that students who get their choice of schools take a bit more ownership. If they get their second choice, or last choice, or somehow do not get their choice, that's a big hurdle for their teachers and parents to overcome. If parents do not get their choice of schools for their students, are they going to be as supportive of their child's teachers? Of the assignments? Are they going to have the same kind of faith in their child's school? I think that the answer is really quite obvious.

The problem with these studies is that the students and families who "lose" these lotteries are no longer like the students and families who "win" these lotteries. There simply is no basis for thinking that their views of their schools are like those of the lottery "winners." In fact, one could quite simply argue that this method of analysis ensures that the "winning" charter school students are being compared to students who did not want to go to the schools they attend.

Obviously, that's not an even comparison.

Issue #2: Peer Effects

We all know that peer effects matter. Research, experience and common sense back this up. If you put a student in a class with a "better" group of peers, s/he will do better than s/he would have done in a class with a "worse" group of peers. The other kids all do their homework, or their parents were more likely to read to them, or they are somehow smarter, or harder working, or bring more cultural capital to school with them, or however else you think "better students" might be defined.

We also know that charter school students are not, as a group, like non-charter school students. That is how the Gold Standard crowd justifies their approach here. So: trying to control for applicant differences, but not controlling for ongoing peer effects? I don't know if that is just lazy or actually dishonest. The importance of peer effects is so well recognized that I tend to think it is the latter, especially because techniques to control for them are so well established.

Of course, there is another way to look at this. From a personal level, if you have a child, you don't care about controlling for peer effects. Actually, you want the effects to be left in so that you can take advantage of them. If charter schools have "better" students, that's a reason to send your own child to a charter school. However, if this analysis is done for policy purposes, to influence policy-makers, then peer effects do matter. If you are thinking about all students, not just the select few who can get into the "better" school, you need to control for peer effects.

Issue #3: Selection Bias on the School Level

The goal of this lottery-based study design is to avoid self-selection bias in the data. However, those who use it do not acknowledge the additional selection problems they create.

The most important problem is that not all charter schools are oversubscribed, so not all charter schools can be included in these studies. This wouldn't be a problem if we had good reason to believe that a random selection of charter schools were included, but that is obviously not the case. Clearly, the "better" charter schools are far, far, far more likely to be oversubscribed than the "worse" charter schools. This biases the sample rather severely towards better charter schools.

Unfortunately, the sample bias problem doesn't stop there.

A really strong traditional non-charter public school is not going to lose a lot of students to a simply above-average charter school. In order to be oversubscribed, a significant number of students and/or families have got to believe that the charter school option is superior to the non-charter public school option, which suggests a level of dissatisfaction with the local traditional public schools. This biases the sample towards inferior non-charter schools.

Issue #4: Generalizability

The hardest thing in educational research -- and perhaps research overall -- is to be able to generalize one's results to the broader population or wider world. And yet, that is usually the end goal of policy-oriented research.

These kinds of lottery-based studies only include the kinds of students and families that apply to charter schools in the first place. Even if the previous issues could be corrected, how can one know that other sorts of students and families would see the same benefits? The fact is that different populations might benefit less or more from going to a charter school. It is simply impossible to know from this kind of study. Of course, if you are only concerned about benefitting the kids of families who already opt for charter schools, then this is not a problem. But if you aim to help a broader population than that, you need a better methodology.

These generalizability concerns also apply to schools. Oversubscribed charter schools might well be better than average non-charter public schools, and I do not really question whether they are better than their local traditional alternatives. But on a policy level, we need to be concerned with charters more generally than that. If we raise or lift caps on charter schools, or approve new charter schools, we have to expect an average charter school to result, not an exceptional one. But these studies really tell us nothing about the majority of charter schools that are not oversubscribed. Nor do they tell us anything about the relative quality of non-charter public schools that lack charter school alternatives.

*****************

I understand the desire to find a Gold Standard for educational research. But simply grabbing that label because a methodology has some resemblance to biomedical research is not good enough, despite what Prof. Caroline Hoxby may claim. Moreover, the popular press really must do a better job of examining these claims critically, rather than cheerleading for them like this.

This, of course, means that researchers, journalists and the rest of us must be sure to take a more thoughtful stance than has become our habit.

Engagment is not enough

School is starting, in most places either last week or this wee. Like any other sort of new year, it's a time for resolutions, for setting new expectations and resolving to do the right thing.

The president spoke to the nation's children, or tried to, with this in mind. Time to think about hard work, finding your strengths and persevering through the challenges, he said.

Children are not the only ones who begin a new year around labor day. Parents, too, begin a new cycle. They see their children off to school, to a new grade, and know that themselves are beginning a new year. For many of them -- perhaps most and eventually we hope for all of them -- one of their own issues each year is figuring out how to support their children's learning. Buying school supplies is easy. What about the rest?

Obviously, parents can help their children with their homework. They can set aside a quiet place and time for them to study. Is there more?

Well, traditionally, parents would ask their children "How was school, today?" Maybe they'd ask "What did you learn in school today?" Trite questions, to be sure, but powerful nonetheless. How better to signal to their children that they care about education and are invested in their learning than this sort of thing?

Of course, those questions get old. Kids hear them too often, and they don't feel genuine. Parents feel silly asking the same question every day. And engagement in this sort of important conversation fades.

With all of this in mind, Will Richardson has written about the kinds of questions he hopes to ask his children this year, and offer 13 examples. His readers have offered dozens more in the comments. Together, they make an interesting collection.

But I'm not satisfied.

In fact, I am troubled.

I don't have a problem with any of the questions, I don't think. But taken in their totality they seem a bit thoughtful and random. Yes, they are good questions, but so what? What is goal of these questions? What are they meant to accomplish?

You see, I think that one of the great problems with American schooling is the muddle. We do many things, with not real focus on plan on the deeply meaningful long term goals. We do little things to satisfy everyone, but never really focus on what we want to accomplish. Taking a bunch of ideas and throwing them together does not necessarily make for a great combination. Too many cooks can spoil the soup, right?

I think that we see this in technology. Most tech companies design products with lots of features, to meet the needs of a broad swath of the potential market. More features, they seem to believe, leads to broader appeal. But like the big swiss army knife, american schools has gotten unwieldy and hard to use.

Apple has taken a different approach with its products. They often have fewer features, but a much greater sense of cohesiveness and purpose. More integrated, and even thoughtful. Never the only way to do a great product, but almost always a great product.

So, I am not saying that any of Will's questions are poor, or even any from his commenters. But I really do not know what they are seeking to accomplish. These questions have different goals and assumptions behind them. And my real issue with his post and the comments is that they are mindless of these issues, of the goals and assumptions that their questions represent.

What Did You Create Today?

What did you learn about fairness today?

What will you now do tomorrow because of what you did today?

What was the coolest/most interesting question someone else asked today?

These are great questions, as are the others in will's post and the comments. (Really, they are good questions. Go and read them.) I am not challenging that. But they have different goals. When it comes to education, we really do need to be mindful of goals, and make sure that what we do meets the goals we have in mind.

So, if parents goals are merely to have engaging conversations with their children about their school days, any of these questions might do. Switching among them might work, too. Yes, that is a worthy goal. But if parents want to reinforce some aspect of learning or schooling, encourage particular kinds of behaviors or habit of mind, selection from among these questions requires them to be more thoughtful.

And the exercise of uncovering the assumptions and goals of these questions would be a good way for any parent or PTA to get a hold on some of the hardest and most important questions in education.

My Senator Died: What Ted Kennedy Meant to an Educator

My senator died. I think that a lot of educators can say that today. Only a small fraction of us ever voted for him, or have ever lived in Massachusetts, but Ted Kennedy stood for what we stand for.

Sen. Kennedy served for so long -- 47 years -- that from the time I first got a clue about what the senate really was and what senators really were until today, he has been my senator. When I really became politically aware as a teenager, the President did not reflect my views or values. It was important to me to have a national voice that represented me, and Senator Kennedy was that national voice. I only had the chance to vote for him once, but he has always been my senator.

*****************

Looking at the education landscape today, it is easy to conclude the NCLB has a stronger impact on schools than any other federal legislation. It is easy to say that Sen. Kennedy got rolled when it came to NLCB. He thought that the money it authorized would actually be budgeted and spent, but the Bush administration's budgets only called for a fraction of it.

Obviously, there are lots of aspect of NCLB to hold against President Bush, and also against Senator Kennedy (D-MA) and Representative George Miller (D-CA). Without their credibility and their cover, this bill would not have passed.

But we should keep in mind the positive aspects of NCLB, too. Senator Kennedy did not like the whole bill, but he thought that the attention it would bring to urban education was worth its costs. In hindsight, I think he was right.

NCLB has required subgroup reporting of test scores. Though statistical rigor seems to run afoul of moral considerations, this has been a revolutionary change in the way we examine schools. And it is federally mandated!! Every school must examine how its minority students perform, and report it publicly. Every school must report how its low income students perform, and report it publicly. No longer can schools can schools hide poor performances of some groups with superior performances by others. Schools are now have real incentives to pay attention to all of these groups.

Does anyone think that this is going away?

Senator Kennedy was right, I think. We are paying more attention to the performance of minority and low income students than ever before. That is no small thing. The federal spending on education is at an all time high, and its hard to imagine that being possible with NCLB. That's no small thing.

Furthermore, for all the problems with NLCB's "highly qualified teacher" provision, the fact that we are even having a debate about it is enormous progress.

There is a lot of stuff to dislike about NCLB. But it has prompted some revolutionary shifts in how we, as a nation, talk about students in schools. And that has been wonderful.

*****************

However, Senator Kennedy was not my senator because of NLCB. He was my senator long before NCLB, and I spent years thinking quite a bit less of him for NCLB. He was my senator because he stood for things that I believed in. Most fundamentally, he stood for fighting for those unlike himself, less fortunate than himself, and doing so simply because it was the right thing to do.

I think that all educators understand devoting our lives to helping others. In fact, educators understand working to help those with less power and privilege than we do. Children inherently have less power than adults, and helping them does not help us to get ahead. None of us expect the favors to be repaid, even if the love and compassion is. We do it because it feels like the right thing to do, because it means more to us than maximizing our own power and position.

Senator Kennedy did not have to be the senator he was. He stood forward to fight for what he believed in, making himself a target for criticism and mockery -- when he knew that he had major personal failings to be mocked. He took unpopular stands because they were right, not because they would help him or his career. Most teachers know about fighting to do the right thing for their students, even when it makes their lives harder.

*****************

Last, I want to address Senator Kennedy's longevity.

He was in it for the long fight. There were some tremendous victories along the way, but they came after a lot of work. He was not looking for silver bullets or miracles, and he knew that that each legislative victory was but a single step in a very long road.

NCLB was an important step, but just a step. If we had him in Senate for the rest of his term, we would see him address its strengths and and its weaknesses.

We need to remember this. The big problems that Senator Kennedy worked his whole career to address -- education, poverty, the rights of the disempowered -- are not solved easily, or perhaps even ever. Rather, they are addressed as best we can today, so that we can address them even better tomorrow.

Perhaps this is an easier lesson for those of us who work in education than outside of it. We know that the our students' educational journeys last much longer than our own time with them. I hope that those who do not work in schools can learn this, too. Especially when trying to address education.

Incentives and TED

Do you know about TED?

TED is this annual conference where people make these really great 20 minutes presentations. They come from all walks of life, but they are all smart. The presentations are intended for a small general audience, not a group of experts. TED's tagline is "Ideas Worth Spreadings," and they generally are.

Fortunately, the folks who run the conference have posted all of the talks online. For free, and going back years!! What's not to love?

TED is certainly more thoughtful than 99% of what you'll find out there on the Interwebs. High quality speakers with high quality content.

While I was thinking about my previous post, I came across a TED talk on the same subject (i.e. incentives and performance). Of course, it is better than anything I could write. So, go watch it.

Understanding Incentives

There is a big debate in education right about incentives. Sometimes it is explicit, and sometimes it is implicit, but it is a part of quite a few of the button issues and policy questions. There are those who believe that intrinsic motivation is key, and others who believe that extrinsic motivation is a powerful and untapped force that we need to make better use of. The former group believes the opposite, that use of extrinsic motivation is powerfully destructive and we need to beware of it.

What should we do about this difference? How do we evaluate the question, so that we can made pedagogical and policy decisions?

One way, the least thoughtful way, perhaps the most common way, is to go by our gut. That really just means going by what we already believe on the question, without examination or consideration. The problem with this approach is that no one ever learns anything, and there is no way to convince anyone -- not even ourselves -- of anything new.

I am a big of another way: the thought experiment. I like other approaches, too. But the thought experiment strikes me as the essence of the more thoughtful approach. Wikipedia tells me that its origins are different than I had thought, and that my understanding of it might not be archetypal. But I'll stick with my approach.

Think about what you are proposing. What would that imply down the road? What does it rely upon or assume? If it were true, what else would have to be true? How would you test if it were true (that's more archetypal) and has that test already been done? Does any of this contradict what we already know to be true?

I suppose that this a mathematical approach to proving something -- not surprising, considering that math was my first discipline. Work through the proof, and if you find a contradiction, the supposition must be false.

This has usually been my approach to merit-pay for teachers. For this to work as a reform, teachers would have to motivated by financial reward. The problem is that the given the widespread belief that teaching is a poorly compensated job -- what that is true or not -- it is hard to believe that the kinds of people who go into teaching are likely to be motivated by financial rewards. In fact, those that are would already have chosen another profession, for precisely that reason. Some supporters then counter that this would attract those very people to teaching. So, I ask whether the level of financial rewards that people discuss for merit pay would be sufficient to attract such people. I mean, are they likely to attracted because of the size of merit-based bonus, or the size of total compensation? Obviously the latter, right? So, how large would these bonuses need to be such that, when added to the base salary, they attract such people to the profession?

I like this approach, but it is not the final answer. A though experiment can be quite useful, but it does not always lead to a final answer, and even when it does others may not be convinced.

Luckily, there often are real studies, or even real experiments. When it comes to motivation, the work has already been done. For example, my attention was recently brought to a paper by a bunch of researcher who have nothing to do with educational policy or pedagogical issue. Actually, the paper is published by the Federal Reserve Bank of Boston, and is by a bunch of business school professors.

Ariely, Gneezy, Lowenstein & Mazer (2005) looked the impact of external financial motivation on performance. They considered tasks with requiring varying degrees of concentration, creativity and problem-solving. They found that for high rewards actually tended to degrees performance. That's right. Decrease.

First, let me point out that that is a problem. Teaching requires a lot of creativity and problem-solving. If high contingent financial rewards hinder performance with such tasks, we would not want to bring to to our schools.

Second, this experimental finding really poses a problem for offering merit-pay at a high enough level to attract the sorts of people motivated by financial rewards. It is like being between a rock and hard place. Attracting them takes a lot of money, but a lot of money decreases performance.

Third, I want to add a bit more discussion. Why is there such fervent belief in the power of of financial rewards for performance? I can think of two ones, one which the research literature addresses, and one which it does not -- or at least I have found it.

Clearly, in some situations extrinsic rewards can make a difference. When it comes to effort and narrow concentration, we often see an impact. Simple tasks, repetitive task, physical tasks. All of these can be very responsive to extrinsic motivation. But creative and problem solving tasks? The kinds of things where we need to wider our focus, in order to find solutions -- or at least next step? The research has made clear the extrinsic motivation does not really help there, and can even hurt.

So, why do we believe in it so? I think that there's another issue. I think that it might work in some cases. I think extrinsic motivation (i.e. rewards, prizes or even avoiding punishments) can help to identify the absolutely best performers -- at least best at performing when in such environments. But the fact that they might even be able to raise the performance of those who do best in such environments does not actually make them a good idea. If they make some small fraction of performers a bit better, but make most performers worse, that is a problem. Obviously, there are some configurations of that dynamic where the tradeoffs are worth it, not that is not always the case.

So, the more thoughtful approach means we consider not just the benefits, but the costs as well. In the case of merit pay for teachers, is it likely to help most students or help those student most in need? Or will it help a random minority of students at the expense of the rest?

From Findings to Implications

One of the most significant reports to be published this year is from the Center for Research on Education Outcomes (CREDO). Their National Charter School Study carefully examined achievement gains by students in charter schools to those in traditional non-charter public schools.

This report is so significant, in large part, because of the breadth of their sample. They included roughly 70% of charter schools students in their study, students spread across many states with different charter school laws. Furthermore, they look at data in the level of individual students, comparing each charter student to another student from their same feeder school -- one matched on grade level, gender, race/ethnicity, free or reduced price lunch status, English language learner status (ELL), special education status and prior test score on state achievement test (p. 16).

This is good stuff. Yes, it is limited, but all researcher face practical limits. In this case, they could not address home or family factors (e.g. number of adults in the home, parents educational level, etc.), because legally they could only get access to certain data. From what I can tell, they did a great job.

My concern stems from how people read their findings and extrapolate too far to implications, something that happens quite often.

In this case, the reachers found that "Two subgroups fare better in charters than in the traditional system: students in poverty and ELL students." I do not question their methodology or their quantitive analysis. However, I think that they are thoughtfully presenting the limits of their sampling strategy -- limits imposed by the nature of the issue they are addressing.

One critique of charter school studies is that they do not account for self-selection bias. The authors of this study acknowledge the issue, and directly address how they attempted to deal with it. But I think that the problem is a bit more nefarious than they let on. Self-selection is only a problem if the self-selectors might be different than the non-selectors in other relevant ways. When it comes to charter schools, this might well be the case. I believe that charter school self-selecting families have a greater commitment to education/schooling than the general population, and a greater sense of agency (i.e. "something that I can do can make a difference"). Transmitting these sorts of values to their children is one of the many ways that families help children to succeed in school.

I think that this difference is especially pronounced among families in poverty and families of English language learner students. When these families step up and find what they believe is a better school for their children -- regardless of whether or not they are making a good decision -- they are modeling important behavior for their kids. This is not to say that demographically similar families who do not choose charter schools necessarily lack these traits. Not at all. Rather, I believe that they are significantly more common among those self-selectors.

I truly believe this finding (i.e. that ELL students and students from poverty do better in charter schools) is merely the self-selection issue at work. Unfortunately, the report's authors fail to address the issue of self-selection in their own implications section (pp. 49-51).

Before this report game out, I believed that charter schools would -- in the long run -- perform just about the same as traditional non-charter public schools. However, various findings in this report have convinced me otherwise. I do not know if leaving out the remaining effects of self-selection was merely an oversight on their part, or some kind of misguided attempt to boost their own credibility by providing supports for charter proponents' claims. But I am concerned that those proponents will latch on these particular findings without acknowledging the real dynamics at play.

And so, I wish that CREDO had presented their implications a little more thoughtfully.

Recognizing a Difference in What Is Being Talked About

It is nice when the world or the Internet provides a perfect example.

NPR has a podcast and a blog on the economy called Planet Money. In yesterday's podcast, they had a little debate about whether financial innovation over the last twenty-give years has been, on balance, good or bad.

Starting at about the 21:30 mark, the participants start to address the question of what they are talking about. At around 22:25, one of them goes so far as to say that by his opponent's definition he would have to agree. When they begin to address their different understandings of the question at hand, we start to see the real disagreements between them -- and the larger areas of real agreement.

Of course, it's NPR, which has has made its reputation on being more thoughtful.

What Are You Talking About?

Quite a few years ago, I got into a friendly argument with a friend's much older husband. Well, he and I had quite a few friendly arguments, but this one was fairly specific: who was the best running back in NFL history?

I'm a Barry Sanders guy, though I respect the Jim Brown argument. I don't care for the Emmitt Smith argument at all, though I'm a bit more sympathetic to the Walter Peyton argument. My friend's husband did not advance any of these candidates. He argued for Gale Sayers.

I was floored! How could he say that!?

Well, there were two big reason that I did not think of. His first reason, the one he consciously espoused, was that Gale Sayers was the best return man in NFL history, and that meant that you got two great players in one package. This was a hard argument for me to make sense of at the time. I thought that this guy had gone senile or something. We were not talking about that, so why was he bringing it up? I found his point to interesting, but largely irrelevant. He truly believed his argument, and did not see any way that anyone could answer it.

I think that this sort of situation if found throughout our education debates, both on the policy level and on the pedagogical or curricular level. We think we know what we are discussing, but it turns out that the participants in the discussion/debate/argument really are talking about different things. Neither side is convinced by the others' arguments because they are truly engaging on the same topic.

In my example, I was arguing about who did the best job at running back, and he was arguing about which guy who played running back was the best player overall. I've got to say, each of those seems a valid interpretation of the original question ("Who is best running back in NFL history?").

In our educational debates, sometimes those differing interpretations of the questions at hand are both truly valid, and sometimes they are not. But how often do people stop arguing against their opponents to figure out what those opponents are arguing for? I believe that the best ways to counter someone's arguments always begin with understanding them, that requires understanding their goal -- in this sort of case, understanding their interpretation of the question.

Obviously, there is lots of disagreement about the aims and goals of schooling. I think that people would generally agree, however, that preparing students for their lives as citizens and members of society are among them, and that that includes preparing them to understand and participate in public debates. I believe that that is not a controversial idea. Well, I wonder about how well we can do that if we are not so capable ourselves. Can we teach habits that we do have, or are not even inclined to use on a less-than-habitual basis?

I am not sure about the answer to that last question. More thoughtful schools, however, would address that question for themselves.

Who can pass tests?

There is a great great book call Tinkering Towards Utopia, in which the authors look at the history of school reform in the United States. One of the many interesting things they point is the the resiliency of the "grammar of schooling." That is, we all know what school is, what it looks like and what it feels like. Something like thirty students in desks, probably in row. One teacher, probably in the front of the room. A blackboard and chalk. Perhaps some sort of overhear projector. Chapters or units. Homework. Tests. Final exams. 95 is an A, and 55 is an F. Bells ringing to signal motion.

This is an idea I go back to a lot. I wonder how much of this is truly worthy, and how much is merely deeply ingrained habit.

For example, what about those tests? Be they unit or chapter tests, or final exams, what are they for? What do they mean? (To be honest, and I much more a high school educator than an elementary school educator, so the following should be taken in the context of high school life, high school students and high school tests.)

Do you think that you could go back and pass all your old final exams? What about the unit tests? Probably not, right? So, if you could not pass them later in life, what was the significance of giving them those tests and grading them as they did? And as we still do! In the long run -- and I hope the we think about education in the context of the long, in addition to thinking about the short and medium run -- what was the purpose of those tests? What does it mean that we demand students perform at a level that their parents cannot? What does it mean that we know that these students will only have this kind of mastery or skill for a short while?

It goes even further than that. Of course, the Algebra teacher can pass all of the algebra tests. But can s/he pass the Calculus test? Can the Calculus teacher pass the geometry test? Can s/he pass the French Revolution test?

What does it mean that we demand that students do things that their teachers cannot? And what does it mean that the students know probably know this? (I mean, how many students ask their English teachers for help with the math homework, or their science teachers for help with their social studies homework?)

I don't have answers to these questions, though I've been thinking about them for over twenty years. I remember when I first starting thinking about how I might be able to assess my students in a way that would better tap into their longer term retention, before I realized that such a thing would cause outrage from students, parents, colleagues and supervisors. This was, of course, decades before I reading Tinkering Towards Utopia.

I wish there was a place in our debates about education policy to address these sorts of questions. I deeply wish that there was a place in our pedagogical debates, teacher pre-service training and/or professional development to discuss matter likes this, and this one in particular. It would indicate a more thoughtful approach to education, one capable of truly revolutionary improvement.

Because It's Hard

Did you see Ami Novoryta's commentary in Education Week? It is about selection of school district leadership, and it is not self-serving.

It is far more thoughtful than most commentaries you read about what districts should do when selecting superintendents.

Wrong-Headed Comparisons to Other Professions

A favorite tactic in debates or discussions about teachers is to compare teaching to other professions. Wrong-headed, ill advised and perhaps even flat out dumb comparisons abound. Today, I got into an argument with Eric MacKnight on Will Richardson's Weblogg-Ed about a bad comparison, but it is hardly the first time I have seen this, or the even the thousandth.

For example, some like to point out that that only teachers are unionized, among the professions. Alas, this is just wrong. Unionization depends far more on your industry and/or employer than whether you are a member of a profession. Many lawyers and doctors who work in the public sector are unionized, example. More importantly, one should acknowledge that nurses and airline pilots are generally members of unions. So, the next time you hear that unions are incompatible with high quality professionals, just respond "Airline Pilots."

Of course, it is not just teacher bashers who make poor comparison to other professions. Have you ever hear teachers complain that they are not being treated like professionals? For example, strictures on what or how they teach are not befitting their professional status. What they seem to miss is that they work for someone else. Young attorneys are closely supervised and their work is checked. Clients can dictate strategy and tactics, even overriding their attorney's professional judgement. Doctors have layer upon layer of oversight, both during their residency periods and later (e.g. insurance companies who second guess their treatment plans).

I am all for comparisons to other professions. I think that we can learn a lot by looking at parallel situations, because it is often quite difficult to get a good perspective on our own. We can learn from the experiences of others, both stealing the good and trying to avoid the bad.

The problem, however, is when these comparisons are based on fantasies. Some of them are literally based on fantasies, such as when people's knowledge of how law firms work are based upon what they've seen on television. Others are a bit more personal. But all of them could be addressed simply by talking to people to ask them about their work. All of them could be addressed by doing some research or some reading.

Instead, however, too many people love the facile and uninformed comparison.

Being more thoughtful means interrogating your own thinking and your own examples, comparisons and analogies. It means trying to verify your claims with people who are in a position to know, and looking to see if there already exists clear overwhelming counter-arguments. We want to encourage our students to do this, no? Whether it is a mathematical argument, a historical argument, a literary argument or a scientific argument, we want them to be more thoughtful than that.

So, why can't the adults who argue about to improve education do the same?

Social Promotion: Research vs. "Common Sense"

Unfortunately, many of the most public policy debates in education come down to research verses common sense. But common sense is like truthiness, in that it does not have a lot of regard for what is actually true or require any sort of real basis for belief. Being more thoughtful, in this sort of case, means looking beyond "common sense" to figure out what we really need to do. The issue of the day in New York is social promotion.

First, what is "social promotion"? Well, it is the practice of promoting students to the next grade, even though they have not "passed" the last grade. "Retention" is the policy of holding back kids who did not pass a grade to repeat the grade again. There's a lot of intuitive appeal to retention policies. A commenter going by the name insiderknowledge on Gotham Schools -- my favorite education blog -- addresses most of the common objections to social promotion.

What message is being sent to failing students who are promoted to the next grade? That we care more about your self esteem then your education? Why even bother with the diploma at the end of 12th grade if one has to do almost nothing to earn it? Students earn grades and those that EARN passing grades should get promoted those that don’t shouldn’t plain and simple. This country started its decline in educational performance when it got into the self esteem management business. Failing is the natural way that we learn. When we fail we learn what not to do or what to do that will make us successful.. i say that those who eventually drop out as being kids with very low character. And that’s what this nation has become. A nation of low character quitters who toss in the towel when they are not instantly gratified whether they deserve it or not. If we recognize after a certain level that a student simply cannot meet the criteria academically to pass to the next level we should be steering them into vocational training so that they have a career to look forward to. That’s the real reason they drop out.. Not everyone is cut out for academia but we have this silly notion that everyone has to go to college to get a liberal arts degree.

Not to pick on insiderknowledge, but this is a classic "common sense" kind of response. Social promotion is not a high school issue as much as it is elementary and junior high school issue. I have no reason to believe that insiderknowledge favors a vocational track for 4th gaders, but s/he has clearly conflated potentially wrong-headed expectations about universal college attendance with the issue of social promotion. On the substance of this issue, s/he worries about the "message being sent," rather than what actually happens to children.

The other side of this debate take quite a different tone. I do not know of anyone who likes the idea of social promotion -- after all, it is rather counter intuitive. However, actual research has clearly shown that retention does not work. Kids who are held back drop out in much higher numbers than those who are not. Kids who are held back twice are overwhelmingly likely to drop out before graduation. The sense I have gotten from the research is that if they are not held back, they are less likely to drop out. (That is, this is not just a self-evident find of "kids who do poorly in school drop out more"), and holding them back actually increases their chances of dropping out above and beyond what doing poorly in school might predict. There is other research out there, too, about other negative impacts from being held back, thinks like violence, teen pregnancy, drug use, etc. -- though the causal link is less clear in those cases. In fact, it is virtually impossible to find anyone who has actually researched student outcomes and how this stuff actually works who is against social promotion policies.

Pedagogically, I believe the only question should be "Does repeating grades help students more than the dangers caused by holding them back." If you believe that it does, then you have must believe that repetition of the same material, taught in the same way, at the same pace is going to make a difference. Richard Elmore calls this the "more, louder" approacher. When I teach and my student does not understand something, I try to find another way to explain it. I believe that if the long schooling endeavor is to worth anything to students, it is in part based upon repeating ideas in different ways, manners, contexts and usages. To put this more plainly, if it didn't work the first time, why do you think that it is going to work the second time?

(Of course, insiderknowledge makes an important point: what is the value of a high school diploma if we do not know what kind of mastery it might indicate? This gets to a broader issue, I think. That is, what do our external signal of educational attainment (i.e. grades, transcripts, courses, diplomas) actually indicate? In the future I will address this a bit more, but for now suffice it it say that social promotion is the least of our problems in this area.)

So, the research is clear. Retention does not work for students, even if it does "send a signal." Social promotion helps kids -- and not just their self-esteem. Social promotion contributes to kids staying in school longer. It might means that some kids graduate with less mastery that the rest, but it seems that they graduate with more mastery than they would have had had they they dropped out years earlier.

You see, the false debate is one that assumes that the alternative to social promotion is a retention policy that assures the repetition of a grade leads to mastery of the material that enables students to complete the rest of their education just like the other kids. That is simply not the case. Those against social promotion are in favor of fantasy, and do not pay enough attention to reality to even be aware of it. The reality is that we do not have sufficient special programs to grade repeaters to help them differently the second time.

So, which approach to such issues should we take? Should we go with our gut? Or should be look at the research and realities of the programs on the ground? Surely it is easier to go with our guts, but don't our children deserve a more thoughtful approach?

Vander Ark Misunderstand the Market for Assessments

I met Tom Vander Ark* once. He seems like a nice guy. He's clearly no idiot, and wants good things for education. But I have the feeling that I will write a fair number of posts on this blog describing why he doesn't know what he is talking about.

* Mr. Vander Ark, in case you do not know, used to run Bill Gates's educational philanthropy. When you have that kind of money to give way, people in this country care what you have to say. I mean, Mr. Gates was invited to address the National Governors Association on the topic of high school reform when he has never studied, worked at or attended a public high school. Vander Ark is responsible for the money wasted on creating a whole bunch of small schools. Now, he says, "Small is not a panacea" and admits they need to "shift gears" from small schools to another reform idea. Oops.

So, now Mr. Vander Ark is projecting how national standards will create a new marketplace for educational assessment. So, Mr. Vander Ark, can you meet me on camera three to discuss this a little bit?

*****************

One potential benefit of national standards could be the development of a new marketplace for assessment systems.

Ummm...Tom, do you think that there's not already a market for assessments? What do you think ETS, Pearson, McGraw-Hill, Harcourt and the rest do? Did you think that they just publish books? In fact, most educational publishers are also assessment publishers. There is already quite a market, a very profitable one, from what I hear.

Rather than the oppressive, expensive, end of year bubble sheet drill that states currently impose on schools, states (or consortia) could accept proposals for alternative assessment systems aligned to common standards. Many of the proposed alternatives would incorporate online and adaptive assessments and performance demonstrations. This would encourage a generation of adaptive content and proficiency-based school formats.

Tom, the very reason why "bubble sheet" tests are used is because they are relatively inexpensive. They are cheap to score, and easy to report. Every other design is at least as expensive to score and report, while being more expensive to design and to give. Of course, there are very technical matters of reliability, which make performance demonstrations quite problematic -- much to my own chagrin.

Furthermore, how would national standards accomplish any of this, if California or Texas state standards did not? Do you think that either state lacks sufficient schools or districts to create a market for better assessment, if size was the determining factor? I mean, I understand why Wyoming or Alaska might not be worth their while, but California? We have had state standards for a long time, even if we have not had national standards. In all that time, we've not seen this revolution in assessment that you seem to be predicting.

*****************

I'm not sure that I disagree with Mr. Vander Ark's goals for future improved educational assessment. But what disturbs me is how he does not appear to have any idea how we got to where we are, or how current efforts might or might not impact a particular issue.

Not all good ideas are feasible. Quite disturbingly, there are many good ideas that we simply cannot achieve. To take an example from another area, President Obama cannot propose a "single payer" healthcare model because we cannot get there from here. Start from scratch, and it might be a good idea, but given our history and the status quo, it simply is not achievable.

And so, if one wants to make serious proposals -- or even suggestions -- one must think beyond "Wouldn't this be a good idea."

This is a much bigger problem than Mr. Vander Ark, however. I believe that he is woefully short of experience and education in this field, as his bio on his firm's web site attests. He's a businessman who might be able to run an organization, but what does he really know about classrooms or education? He spent some time running a 22,000 student school district -- which is far far far far far more than many of the loudest voices in these debates -- but that is high level management stuff. Of course, many of my problems with Mr. Vander Ark stem from disagreeing with his actual proposals and suggestions, regardless of the degree of his experience or education. However, if he -- and others -- actually had relevant experience and/or education, they might might be triple check my own thinking. But they so rarely do.

*****************

Education is a serious enterprise. Truly great ideas, innovations, reforms or even suggestions require real thoughtfulness -- which in turn often requires real preparation. The idea that smart, hard working or successful people from other fields or sectors can come into education and fix things without putting in the long and deep thinking time required to even understand what is going on is not only insulting, it is destructive to the futures of those we are all trying to help.

And it is the antithesis of what I mean by More Thoughtful.

What do I mean by "More Thoughtful"?

Is this more thoughtful? I hope so. I hope that this blog is more thoughtful that most of what you read about education or education policy. But that is not really what the title of my new blog refers to.

Title refers to the need generally for society to be more thoughtful, and particularly the need for more thoughtfulness in education.

Our schools need to prepare students to be more thoughtful, and to do this we need more thoughtful teachers. Teachers need to think more about what they are doing in the short term, especially how it fits into their longer term goals for their student. Teachers should think more, individually and as a group, about how their long term goals fit together and how those longer term goals taken together comprise a meaningful education.

Of course, shifting our teacher corps to a more thoughtful stance requires more thoughtful school leadership. I do not mean to imply that such a thing is easy. School leaders have inundated with responsibilities and cries for attention from all areas and every imaginable constituency. So much falls to school leaders that it is hard for them to have time to think, let alone to be thoughtful. But we need our leaders to lead, not just run things and respond to crises. If we are going to have more thoughtful teachers, their leaders need to model this behavior, encourage it and reward it. We need more thoughtful school leaders.

The chain goes on. How do we get more thoughtful school leaders? Well, their supervisors must also be more thoughtful. They must model, encourage and reward thoughtfulness. And so on, and so on.....

Obviously, high quality schools are not simply a product of thoughtfulness. There are a lot of technical issue, concrete know how, experience, habits and other factors that contribute, as well. I will address many of those things in this blog, too. In fact, most of what I address will be fairly concrete, and might appear to be removed from this idea of thoughtfulness. However, it will always be motivated by my ultimate concern for thoughtfulness both as a goal and a means in education.