The Case For Standardized Testing (Part II): Fundamental Hollowness of Accountability

Inspired by Kathleen Porter-Magee of The Fordham Institute, I present a series on the remarkable weakness of the case for standardized testing in our education policy and education system. You can read part one here. 

The Case For Standardized Testing: The Fundamental Hollowness of "Accountability"

So-called "Accountability" in education is about a particular formulation. The idea is to give educators, schools and or/districts control over the means and methods, but to hold them responsible for the ends.

For decades, we looked at the inputs into schooling. We looked at funding. We looked at teacher qualifications. We looked at textbooks. We looked at curriculum. At management strategies. We looked at what educators were given and/or tried to do. But, so the argument goes, we did not look at what they accomplished -- or failed to accomplish.

The so-called "Accountability Movement" has been an attempt to shift attention from inputs to outputs. Educators, schools and districts can control the means. But they are responsible for accomplishing the ends.

Whether or not you agree with this approach, it is what we have today. And those ends are examined through standardized testing. So-called "Accountability" policies are testing policies. They depend on standardized tests.

*********************

Nancy and Ted Sizer wrote, "Education is the worthy residue that remains, long after the lessons have been forgotten." I talk about "Lessons worth learning for a lifetime."*

* I don't say it anywhere nearly as well as they did, do I?

**********************

Our standardized tests do not measure the important outcomes of education or schooling.

Think about the great lessons you learned in school. Think about the lasting impact. Think about that great teacher who made a difference in your life. Can any of that be tested with a standardized test?

We should recognize what standardized tests can do, says Ms. Porter-Magee. I agree. They can test individual skills, usually in isolation, without authentic contexts or purposes. The Standards of Educational and Psychological Tests point to the importance of testing one thing at a time, so that you can be sure you are not conflating different things and can be sure what the testee can and cannot do. "Standardized" refers to standard questions in a standard testing environment.

Simply as a matter of good professional practice, standardized tests cannot assess students in authentic contexts and cannot assess their skills or knowledge in the kind of interrelated use that matters most.

What standardized tests can do is access the building blocks of authentic proficiency or mastery. But building blocks are not the goals of education and schools. Building blocks are -- by definition --  the means or intermediate outcomes along the way.

There, so-called "Accountability" policy is fundamentally hollow. It is supposed to examine the ends and goals of education, but substitutes examination of intermediate outcomes for the actual goals. In doing so, it replaces accountability for actual meaningful outcomes with reductive, simplified and dumbed down goals.

By substituting what we know how to test quickly and cheaply for the real goals of education, the "Accountability" movement has betrayed its own basic formula and set back efforts to improve meaningful educational outcomes for students.

I am not saying that those who support the "Accountability" movement have done this intentionally. Rather, it's been a laziness; they have been content with what's convenient. They do not look closely at what tests actually can do, at what current test development practices are, at what the best research and scholarship says about what we know and what we know how to do. They insist that these tests are good enough simply because they are available.

And in doing so, they have lost touch with the basic principle of the "Accountability" movement. They do not even live up to their own formula.

Having gutted their own theory of action, proponents of standardized testing have virtually eliminated any chance they can make a strong case to defend their tests or their policies.

The Case For Standardized Testing (Part I): The NRA Defense

Inspired by Kathleen Porter-Magee of The Fordham Institute, I present a series on the remarkable weakness of the case for standardized testing in our education policy and education system. This is part one.

The Case For Standardized Testing: The NRA Defense

Many people who defend either testing policies, the basic idea of standardized tests or the tests themselves try to differentiate between what they are defending and some other flawed part of the equation.

For example, Bonny Buffington tweeted last week, "Standardized tests aren't the problem. It's the undue emphasis on them that causes the stifling of creativity. #edchat." Ms. Porter-Magee has herself tried to differentiate the tests from testing policy.

I don't buy it. To my ears, that sounds like the old line, "Guns don't kill people; people kill people." The guns line, so attacked, refuted and mocked through the years, doesn't need explaining*.

*Full disclosure: when I was in high school so many decades ago, one of the many pins/buttons I wore said, "Bombs don't kill people, explosions kill people."

The fact is that testing policy depends on the tests we have. The fact is that the test developers know how their tests are already being used, and how the trends in how tests are being used suggest they will be used in the near future. Separating the tests from testing policy is as foolish as trying to say that guns are not relevant to understanding our murder rate or the violence in our society.

I call this line of argument The NRA Defense.

I could also call it the Ostrich Defense, because it is like sticking your head in the sand. But I think the obvious finger pointing, as though pointing makes it true, should be highlighted.

So, when we examine the case for standardized tests, or for standardized testing policy, let's skip past that NRA Defense finger pointing, and examine all of it as inextricably tied together, as it actually is.

Common Sense: The argument against

Weeks ago, I said of common sense:

I'll say it right here. I hate appeals to common sense. I've done a little original research on this term, and I believe that it means "What I already think, and cannot/will not justify with reasoning or evidence." It refuses to engage or critically self-examine. 

But I have more to say about so-called common sense.

First, appealing to common sense has got to be the least convincing argument that anyone can make. Those who already agree that your point is common sense need no convincing. Those who do not can only find your claim that what they disagree with is common sense insulting. I literally cannot  imagine anyone changing their mind or reconsidering someone else's position upon hearing someone else say, "It's common sense."

Second, I'm just guessing here, but I'll bet that, "It's just common sense," is as common a claim as, "It's common sense." This underscores my point that an appeal to common sense is a tacit acknowledgement that the position in question lacks reasoning or evidence to support it; either only common sense is available to support it, or (more likely) the person making the claim is too lazy to figure out what the reasoning or evidence might be.

Third, I do not think I have ever heard anyone say, "That guy has just got more common sense than me," though I have heard many, many people try to point out someone else's lack of common sense. Heck, I don't think that I've even heard anyone laud another person's abundance of common sense. Instead, it is only something whose absence is noted.

Fourth, I came across talk of common sense in my dissertation research. When I asked practicing mid-career high school principals about what they are seeing (or not seeing) when they think to themselves, "That AP [or intern] simply does not have what it takes to be an effective principal" (or conversely that a teacher or AP has got what it takes to be an effective principal), their answers often included, "common sense." As I followed up on this, I came to realize that when they said that someone lacked common sense, they almost invariably meant, Your judgment in the moment doesn't match mine, and I know that I am right and you are wrong.

Fifth, "common sense" doesn't actually mean anything. I asked those principals to unpack what they meant when they said, "common sense." I later took the complete list of concepts back to each of them and asked them which are part of common sense and which are not -- pointing out that they all are good, but might not all be common sense. To no great surprise, their answers varied. On a particular day when I had meetings with two different principals, one said to me that it included all but one of the ideas listed, and the other said it was just one of the elements -- the only one that the first principal said it did not include. If there is that little consensus as to what common sense means, it doesn't actually have any particular meaning.

**********************

Common Sense is what I call a contested key construct (CKC). All CKCs share three qualities.

1. Each is a term for a concept or idea (i.e. a construct) about which there is universal (or near-universal) agreement on its importance. 
2. There is no clear consensus as to the precise meaning of the term.
3. There is a dearth of concern about the lack of common meaning.

I have a lot of concern about the use of contested key constructs. At times, they can help to lubricate discussion by hiding substantive disagreements, and thus help build coalitions to build attention, support or resources. However, by hiding those differences -- usually from central participants in the cause, too -- CKCs end up hampering implementation efforts, act to worsen program alignment and prevent decent evaluation and learning.

Let me give you an example. Imagine that a school district got a big grant to support teacher leadership to support school improvement. Imagine that each school got a decent pot of money, and each used the money in good faith. When the leaders of these schools discuss their efforts, they could be talking about entirely different programs that are based upon remarkably different visions for the roles of teachers in schools. But because they talk about their teacher leadership programs going well and who is most active in them, no one ever notices that they are not at all working on the same thing.

Contested key constructs prevent us from learning from each other. They prevent us from even understanding each other. They give the appearance of meaning, but actually act as ciphers.

***********************

If someone is depending on their experiences to support what they think, they should say so. Then, others can better understand them, and evaluate the relevance of that experience to the issues at hand. Then others can engage with them to understand what they actually think, and why. Then, everyone can examine the validity (or potential) validity of the ideas in question. Calling it common sense, instead, prevents all of that.

But just as importantly, appeals to common sense allow the speaker to let him/herself off the hook. S/he is not engaging in the kind of awareness and self-criticism that leads us to growth, learning and the best ideas. We can pay attention to our thinking and our own expression of that thinking. Each of us can make sure that we are being true to our values, ideals, knowledge and priorities -- but not with claims of common sense.

And so, I do not want to hear about common sense. It's a waste of breath, ink, toner, bits and pixels. It's the antithesis of being more thoughtful.

How Broken Is Teacher Evaluation?

On School Finance 101, Bruce Baker wrote today of some of the indefensible things said by state officials about proposed and/or new teacher evaluation models. He highlighted the idea behind many of them, that anything would be better than the status quo.

The standard retort is that marginally flawed or not, these measures are much better than the status quo. ‘Cuz of course, we all know our schools suck. Teachers really suck. Principals enable their suckiness.  And pretty much anything we might do… must suck less.

WRONG – it is absolutely not better than the status quo to take a knowingly flawed measure, or a measure that does not even attempt to isolate teacher effectiveness, and use it to label teachers as good or bad at their jobs. It is even worse to then mandate that the measure be used to take employment action against the employee.

I want to address the unspoken thinking behind that sentiment. Well, actually, I want to highlight the missing thinking behind that sentiment.

By "missing thinking," I mean that there are some important questions whose answers are assumed, without real examination.

The most important question might be: Is the traditional model of teacher evaluation inescapably wrong-headed and flawed, or is it just implemented incredibly poorly?

Regardless of how poorly we evaluate teachers -- and I think that everyone could agree that there is room for improvement there -- we each need to have a answer to that most fundamental question. Our answer there determines what kind of action we need to take.

So, let me unpack the basic elements of our traditional teacher evaluation system, without delving into implementation details.

• Teachers efforts and practices are evaluated (i.e. not their students' learning)
• Evaluation is based upon expert observation of their pedagogy in action
• Evaluation is performed by their supervisor (i.e. department chair, assistant principal or principal.

Obviously, there is currently great distrust -- even condemnation -- of teacher evaluation, but it is not clear to whether that is because people thoughtfully have concluded that the basic model is inescapably flawed or because they do not like the results of we see today of teacher evaluation programs.

Reports like The Widget Effect and officials like Florida Board of Education member Sally Bradshaw seem to object more to the outcomes of those evaluations than the methods. As Matt Di Carlo pointed out, there is a lot of pressure to give more teachers lower ratings.

If that really is the objection, than the model used for traditional teacher evaluation might not be the problem, or at least might not need to be de-empahsized as it is in newer evaluation policies. (Perhaps greater training for school leaders in the standards and expectations for their evaluations could address the problem.)

And so, these are the questions I would ask of anyone weighing in on teacher evalaution:

1. Do you think that expert evaluation of teacher practice, if done properly, is an appropriate way to evaluate teachers?
2. Do you think that those above a teacher in his/her chain of supervision are the appropriate evaluators of their practice (as opposed to their experts peers or some outside inspector)?
3. Can a teacher do everything well in his/her classroom, but student learning be hampered by outside (i.e. home, community, preparation, etc.) factors?
4. Do you think we can effectively capture/recognize/measure all of these relevant outside factors for each student?
5. How much of a teacher's effectiveness rating should be tied to portion of the standards/curriculum/lessons that we can/have put on the big standardized tests?

Note that these question are not necessarily technical in nature, in that they do not necessarily have definitively correct answers. Of course, greater knowledge should influence people's thinking, and there is room for making use of research in answering each of them. But they also get to people's values and expectations, things that not only should be made explicit, but also thoughtfully examined and considered.

While I agree with Baker that there are many things that would be worse that the status quo, what concerns me most is the lack of clarity in the reasoning behind people's objections and policy proposals. I do not know how to evaluate their proposals or how to think about a widely agreeable solution, as it is not clear what people are actually thinking.
 

The Breadth of Real Accountability

Unfortunately, Accountability in education has taken on a very narrow meaning. The term almost always refers just to the use of quantitative data (i.e. most often test scores) to mete out sanctions. Sometimes it refers to to meting out rewards as well, though that is far less common.

In my research, I have asked high school leaders about the full range of accountabilities they face. I have tried to get a fuller picture of what they face and are held responsible for.

The range is pretty damn broad.

So, below is a very simple typology of categories and sub-categories of accountances (i.e what they accountable for), without the complications of the accountors (i.e. whom they are accountable to). It is not a list of individual accountances, as each sub-category is itself full of different kinds and examples of professional responsibilities. Each of those is itself a group of accountabilities.

For example, even test scores is complicated. Of course, there are the tests connection to our test-based accountability (TBA) policies. But schools (and their leaders) are also held accountable for SAT and AP test performance by one constituency or another. Many states have end of course exams (EOCE's), adding more tests that are not even connected to TBA policies (e.g. New York's Global History and Geography Regents Exam). And test score is just one sub-category under Student Outcomes, just one of 46 different sub-categories across the range.

Everything ·  Everything ·  Help Lift the School System School Offerings ·  Quality Education ·  Academics ·  Athletics ·  High School Experience ·  Character/Affective Education Student Outcomes ·  Test Scores ·  System-Wide Goals ·  Graduation Rate ·  College Application Process ·  Learning ·  Prepare for Future/Skills ·  Students' Futures ·  Guilt Students & Safety ·  Students ·  Relationship with Students ·  Safety/Discipline ·  Students Off School Grounds Teachers and Staff ·  Staffing ·  Support Staff ·  Instruction ·  Supervision

Community ·  Be Part of Community ·  Help Parents ·  Public Relations ·  Support PTA Growth Organizational Context ·  Policy Context ·  Relationship w/ Supervisor ·  Relationship w/Peers Branch Administration ·  Budget ·  Purchasing ·  Miscellaneous To District Offices ·  Physical Plant ·  Cafeteria & Transportation ·  Leave a Smooth Operation Behind Classic Leadership ·  Culture/ Environment ·  Good/Fair Decisions ·  Interpersonal Leadership ·  Keep People Out of Trouble ·  Vision Moral ·  Own Character ·  Pay It Forward ·  Representing Owe An Accounting ·  Owe An Accounting

So, the next time you think that our current accountability policy framework addresses anything other than a tiny fraction of what accountability really means in schools, stop and look at that list again. The next time you heard others talking about accountability, without even a tacit acknowledgement that there is more than test-based accountability, refer them to that list.