The Case For Standardized Testing (Part II): Fundamental Hollowness of Accountability

Inspired by Kathleen Porter-Magee of The Fordham Institute, I present a series on the remarkable weakness of the case for standardized testing in our education policy and education system. You can read part one here. 

The Case For Standardized Testing: The Fundamental Hollowness of "Accountability"

So-called "Accountability" in education is about a particular formulation. The idea is to give educators, schools and or/districts control over the means and methods, but to hold them responsible for the ends.

For decades, we looked at the inputs into schooling. We looked at funding. We looked at teacher qualifications. We looked at textbooks. We looked at curriculum. At management strategies. We looked at what educators were given and/or tried to do. But, so the argument goes, we did not look at what they accomplished -- or failed to accomplish.

The so-called "Accountability Movement" has been an attempt to shift attention from inputs to outputs. Educators, schools and districts can control the means. But they are responsible for accomplishing the ends.

Whether or not you agree with this approach, it is what we have today. And those ends are examined through standardized testing. So-called "Accountability" policies are testing policies. They depend on standardized tests.

*********************

Nancy and Ted Sizer wrote, "Education is the worthy residue that remains, long after the lessons have been forgotten." I talk about "Lessons worth learning for a lifetime."*

* I don't say it anywhere nearly as well as they did, do I?

**********************

Our standardized tests do not measure the important outcomes of education or schooling.

Think about the great lessons you learned in school. Think about the lasting impact. Think about that great teacher who made a difference in your life. Can any of that be tested with a standardized test?

We should recognize what standardized tests can do, says Ms. Porter-Magee. I agree. They can test individual skills, usually in isolation, without authentic contexts or purposes. The Standards of Educational and Psychological Tests point to the importance of testing one thing at a time, so that you can be sure you are not conflating different things and can be sure what the testee can and cannot do. "Standardized" refers to standard questions in a standard testing environment.

Simply as a matter of good professional practice, standardized tests cannot assess students in authentic contexts and cannot assess their skills or knowledge in the kind of interrelated use that matters most.

What standardized tests can do is access the building blocks of authentic proficiency or mastery. But building blocks are not the goals of education and schools. Building blocks are -- by definition --  the means or intermediate outcomes along the way.

There, so-called "Accountability" policy is fundamentally hollow. It is supposed to examine the ends and goals of education, but substitutes examination of intermediate outcomes for the actual goals. In doing so, it replaces accountability for actual meaningful outcomes with reductive, simplified and dumbed down goals.

By substituting what we know how to test quickly and cheaply for the real goals of education, the "Accountability" movement has betrayed its own basic formula and set back efforts to improve meaningful educational outcomes for students.

I am not saying that those who support the "Accountability" movement have done this intentionally. Rather, it's been a laziness; they have been content with what's convenient. They do not look closely at what tests actually can do, at what current test development practices are, at what the best research and scholarship says about what we know and what we know how to do. They insist that these tests are good enough simply because they are available.

And in doing so, they have lost touch with the basic principle of the "Accountability" movement. They do not even live up to their own formula.

Having gutted their own theory of action, proponents of standardized testing have virtually eliminated any chance they can make a strong case to defend their tests or their policies.

The Case For Standardized Testing (Part I): The NRA Defense

Inspired by Kathleen Porter-Magee of The Fordham Institute, I present a series on the remarkable weakness of the case for standardized testing in our education policy and education system. This is part one.

The Case For Standardized Testing: The NRA Defense

Many people who defend either testing policies, the basic idea of standardized tests or the tests themselves try to differentiate between what they are defending and some other flawed part of the equation.

For example, Bonny Buffington tweeted last week, "Standardized tests aren't the problem. It's the undue emphasis on them that causes the stifling of creativity. #edchat." Ms. Porter-Magee has herself tried to differentiate the tests from testing policy.

I don't buy it. To my ears, that sounds like the old line, "Guns don't kill people; people kill people." The guns line, so attacked, refuted and mocked through the years, doesn't need explaining*.

*Full disclosure: when I was in high school so many decades ago, one of the many pins/buttons I wore said, "Bombs don't kill people, explosions kill people."

The fact is that testing policy depends on the tests we have. The fact is that the test developers know how their tests are already being used, and how the trends in how tests are being used suggest they will be used in the near future. Separating the tests from testing policy is as foolish as trying to say that guns are not relevant to understanding our murder rate or the violence in our society.

I call this line of argument The NRA Defense.

I could also call it the Ostrich Defense, because it is like sticking your head in the sand. But I think the obvious finger pointing, as though pointing makes it true, should be highlighted.

So, when we examine the case for standardized tests, or for standardized testing policy, let's skip past that NRA Defense finger pointing, and examine all of it as inextricably tied together, as it actually is.

Common Sense: The argument against

Weeks ago, I said of common sense:

I'll say it right here. I hate appeals to common sense. I've done a little original research on this term, and I believe that it means "What I already think, and cannot/will not justify with reasoning or evidence." It refuses to engage or critically self-examine. 

But I have more to say about so-called common sense.

First, appealing to common sense has got to be the least convincing argument that anyone can make. Those who already agree that your point is common sense need no convincing. Those who do not can only find your claim that what they disagree with is common sense insulting. I literally cannot  imagine anyone changing their mind or reconsidering someone else's position upon hearing someone else say, "It's common sense."

Second, I'm just guessing here, but I'll bet that, "It's just common sense," is as common a claim as, "It's common sense." This underscores my point that an appeal to common sense is a tacit acknowledgement that the position in question lacks reasoning or evidence to support it; either only common sense is available to support it, or (more likely) the person making the claim is too lazy to figure out what the reasoning or evidence might be.

Third, I do not think I have ever heard anyone say, "That guy has just got more common sense than me," though I have heard many, many people try to point out someone else's lack of common sense. Heck, I don't think that I've even heard anyone laud another person's abundance of common sense. Instead, it is only something whose absence is noted.

Fourth, I came across talk of common sense in my dissertation research. When I asked practicing mid-career high school principals about what they are seeing (or not seeing) when they think to themselves, "That AP [or intern] simply does not have what it takes to be an effective principal" (or conversely that a teacher or AP has got what it takes to be an effective principal), their answers often included, "common sense." As I followed up on this, I came to realize that when they said that someone lacked common sense, they almost invariably meant, Your judgment in the moment doesn't match mine, and I know that I am right and you are wrong.

Fifth, "common sense" doesn't actually mean anything. I asked those principals to unpack what they meant when they said, "common sense." I later took the complete list of concepts back to each of them and asked them which are part of common sense and which are not -- pointing out that they all are good, but might not all be common sense. To no great surprise, their answers varied. On a particular day when I had meetings with two different principals, one said to me that it included all but one of the ideas listed, and the other said it was just one of the elements -- the only one that the first principal said it did not include. If there is that little consensus as to what common sense means, it doesn't actually have any particular meaning.

**********************

Common Sense is what I call a contested key construct (CKC). All CKCs share three qualities.

1. Each is a term for a concept or idea (i.e. a construct) about which there is universal (or near-universal) agreement on its importance. 
2. There is no clear consensus as to the precise meaning of the term.
3. There is a dearth of concern about the lack of common meaning.

I have a lot of concern about the use of contested key constructs. At times, they can help to lubricate discussion by hiding substantive disagreements, and thus help build coalitions to build attention, support or resources. However, by hiding those differences -- usually from central participants in the cause, too -- CKCs end up hampering implementation efforts, act to worsen program alignment and prevent decent evaluation and learning.

Let me give you an example. Imagine that a school district got a big grant to support teacher leadership to support school improvement. Imagine that each school got a decent pot of money, and each used the money in good faith. When the leaders of these schools discuss their efforts, they could be talking about entirely different programs that are based upon remarkably different visions for the roles of teachers in schools. But because they talk about their teacher leadership programs going well and who is most active in them, no one ever notices that they are not at all working on the same thing.

Contested key constructs prevent us from learning from each other. They prevent us from even understanding each other. They give the appearance of meaning, but actually act as ciphers.

***********************

If someone is depending on their experiences to support what they think, they should say so. Then, others can better understand them, and evaluate the relevance of that experience to the issues at hand. Then others can engage with them to understand what they actually think, and why. Then, everyone can examine the validity (or potential) validity of the ideas in question. Calling it common sense, instead, prevents all of that.

But just as importantly, appeals to common sense allow the speaker to let him/herself off the hook. S/he is not engaging in the kind of awareness and self-criticism that leads us to growth, learning and the best ideas. We can pay attention to our thinking and our own expression of that thinking. Each of us can make sure that we are being true to our values, ideals, knowledge and priorities -- but not with claims of common sense.

And so, I do not want to hear about common sense. It's a waste of breath, ink, toner, bits and pixels. It's the antithesis of being more thoughtful.

How Broken Is Teacher Evaluation?

On School Finance 101, Bruce Baker wrote today of some of the indefensible things said by state officials about proposed and/or new teacher evaluation models. He highlighted the idea behind many of them, that anything would be better than the status quo.

The standard retort is that marginally flawed or not, these measures are much better than the status quo. ‘Cuz of course, we all know our schools suck. Teachers really suck. Principals enable their suckiness.  And pretty much anything we might do… must suck less.

WRONG – it is absolutely not better than the status quo to take a knowingly flawed measure, or a measure that does not even attempt to isolate teacher effectiveness, and use it to label teachers as good or bad at their jobs. It is even worse to then mandate that the measure be used to take employment action against the employee.

I want to address the unspoken thinking behind that sentiment. Well, actually, I want to highlight the missing thinking behind that sentiment.

By "missing thinking," I mean that there are some important questions whose answers are assumed, without real examination.

The most important question might be: Is the traditional model of teacher evaluation inescapably wrong-headed and flawed, or is it just implemented incredibly poorly?

Regardless of how poorly we evaluate teachers -- and I think that everyone could agree that there is room for improvement there -- we each need to have a answer to that most fundamental question. Our answer there determines what kind of action we need to take.

So, let me unpack the basic elements of our traditional teacher evaluation system, without delving into implementation details.

• Teachers efforts and practices are evaluated (i.e. not their students' learning)
• Evaluation is based upon expert observation of their pedagogy in action
• Evaluation is performed by their supervisor (i.e. department chair, assistant principal or principal.

Obviously, there is currently great distrust -- even condemnation -- of teacher evaluation, but it is not clear to whether that is because people thoughtfully have concluded that the basic model is inescapably flawed or because they do not like the results of we see today of teacher evaluation programs.

Reports like The Widget Effect and officials like Florida Board of Education member Sally Bradshaw seem to object more to the outcomes of those evaluations than the methods. As Matt Di Carlo pointed out, there is a lot of pressure to give more teachers lower ratings.

If that really is the objection, than the model used for traditional teacher evaluation might not be the problem, or at least might not need to be de-empahsized as it is in newer evaluation policies. (Perhaps greater training for school leaders in the standards and expectations for their evaluations could address the problem.)

And so, these are the questions I would ask of anyone weighing in on teacher evalaution:

1. Do you think that expert evaluation of teacher practice, if done properly, is an appropriate way to evaluate teachers?
2. Do you think that those above a teacher in his/her chain of supervision are the appropriate evaluators of their practice (as opposed to their experts peers or some outside inspector)?
3. Can a teacher do everything well in his/her classroom, but student learning be hampered by outside (i.e. home, community, preparation, etc.) factors?
4. Do you think we can effectively capture/recognize/measure all of these relevant outside factors for each student?
5. How much of a teacher's effectiveness rating should be tied to portion of the standards/curriculum/lessons that we can/have put on the big standardized tests?

Note that these question are not necessarily technical in nature, in that they do not necessarily have definitively correct answers. Of course, greater knowledge should influence people's thinking, and there is room for making use of research in answering each of them. But they also get to people's values and expectations, things that not only should be made explicit, but also thoughtfully examined and considered.

While I agree with Baker that there are many things that would be worse that the status quo, what concerns me most is the lack of clarity in the reasoning behind people's objections and policy proposals. I do not know how to evaluate their proposals or how to think about a widely agreeable solution, as it is not clear what people are actually thinking.
 

The Breadth of Real Accountability

Unfortunately, Accountability in education has taken on a very narrow meaning. The term almost always refers just to the use of quantitative data (i.e. most often test scores) to mete out sanctions. Sometimes it refers to to meting out rewards as well, though that is far less common.

In my research, I have asked high school leaders about the full range of accountabilities they face. I have tried to get a fuller picture of what they face and are held responsible for.

The range is pretty damn broad.

So, below is a very simple typology of categories and sub-categories of accountances (i.e what they accountable for), without the complications of the accountors (i.e. whom they are accountable to). It is not a list of individual accountances, as each sub-category is itself full of different kinds and examples of professional responsibilities. Each of those is itself a group of accountabilities.

For example, even test scores is complicated. Of course, there are the tests connection to our test-based accountability (TBA) policies. But schools (and their leaders) are also held accountable for SAT and AP test performance by one constituency or another. Many states have end of course exams (EOCE's), adding more tests that are not even connected to TBA policies (e.g. New York's Global History and Geography Regents Exam). And test score is just one sub-category under Student Outcomes, just one of 46 different sub-categories across the range.

Everything ·  Everything ·  Help Lift the School System School Offerings ·  Quality Education ·  Academics ·  Athletics ·  High School Experience ·  Character/Affective Education Student Outcomes ·  Test Scores ·  System-Wide Goals ·  Graduation Rate ·  College Application Process ·  Learning ·  Prepare for Future/Skills ·  Students' Futures ·  Guilt Students & Safety ·  Students ·  Relationship with Students ·  Safety/Discipline ·  Students Off School Grounds Teachers and Staff ·  Staffing ·  Support Staff ·  Instruction ·  Supervision

Community ·  Be Part of Community ·  Help Parents ·  Public Relations ·  Support PTA Growth Organizational Context ·  Policy Context ·  Relationship w/ Supervisor ·  Relationship w/Peers Branch Administration ·  Budget ·  Purchasing ·  Miscellaneous To District Offices ·  Physical Plant ·  Cafeteria & Transportation ·  Leave a Smooth Operation Behind Classic Leadership ·  Culture/ Environment ·  Good/Fair Decisions ·  Interpersonal Leadership ·  Keep People Out of Trouble ·  Vision Moral ·  Own Character ·  Pay It Forward ·  Representing Owe An Accounting ·  Owe An Accounting

So, the next time you think that our current accountability policy framework addresses anything other than a tiny fraction of what accountability really means in schools, stop and look at that list again. The next time you heard others talking about accountability, without even a tacit acknowledgement that there is more than test-based accountability, refer them to that list.

Incredible Changes in Our Classrooms

"Strawman" is a term from rhetoric that refers to to someone mis-characterizing or exaggerating his/her opponents' arguments so as to make it easier to respond to. It it is low and dishonest strategy, infuriating and all too common. Luckily for those of us who think that the transformative power and usefulness of tablet computing and computer in the classroom is has been massively overhyped, we have Rupert Murdoch; we don't need a straw man.

I learned this morning that Rupert Murdoch (Chairman and CEO of News Corp), who has a new tablet to sell to schools, said two years ago,  "Today's classroom looks almost exactly the same as it did in the Victorian age: a teacher standing in front of a roomful of kids with only a textbook, a blackboard, and a piece of chalk."

This just shows how little he -- and far too many who cheerlead for spending more on computer technology in the classroom -- know about the massive changes to our classrooms in the last 100-175 years. (Queen Victoria reigned from 1837-1901.)

* Now, virtually all children go to school through the middle of their teenage years, not merely the children of the rich.

* Now, most students are educated away from the oversight of the church.

* Now, every student has his/her own textbook. Every students has his/her own textbooks.

* Now, we have ballpoint, gel and other easy to use pens instead of inkwells. We have plentiful pencils. We even crappy erasable pens.

* Now, schools have central heating, and perhaps air conditioning too.

* Now, our schools have running water and flush toilets

* Now, our classrooms have electricity and electric lights.

* Blackboards and chalkdust? Disappearing. Now, we have had dry erase white boards and even fancy digital White Boards.

* Now, even in the absence of fancy digital White Boards, we have overhead projectors.

* Now, we have filmstrips, movie projectors, televisions and DVD players. We have a rich array of audio players, as well.

* Now, we have plentiful supplies of paper. Even our poorest classrooms are far richer in arts and craft materials than those of the Victoria era.

* Now, we've moved entirely past the age of dittos -- which came long after the Victorian era -- and are moving beyond the photocopy era. 

* Now, children are divided into classrooms by age or grade.

* Now, we have so many books that classrooms are expected to have their own, "classroom libraries."

* Now, the norm is 20-25 younger students in a class, or 30-35 older students, compared to the 40-50 or more of the past.

* Now, chairs and desks are not nailed to floor, all facing forward in rows.

* Now, students often face each other, rather than the front of the room. Now, in some classrooms, the basic arrangement is to face each other in clusters of desks or at tables, rather than in rows.

* Now, children of all colors, backgrounds and origins can be found in the same classroom.

* Now, the walls of many classrooms are covered with student work, word walls and/or (hopefully) useful posters.

* Now, we might find multiple adults in the same room, especially when students with special needs are in the class -- students who we would not have seen in classrooms during the Victorian Age.

That's just off the top of my head, and I am not education historian. That is just a list of a few things are are obviously different upon visual inspection. We a visitor to listen, s/he would many other differences.

In Search of a Better Education Policy Advocate Taxonomy

On March 5, Eric Horowitz posted In Search of a Better Education Policy Taxonomy on his blog, Peer-reviewed by my neurons. However, he does not offer a taxonomy of education policy. Rather, he posts a taxonomy of education policy (i.e. "reform") advocates. What he really does is try to put advocates into different groups, but without offering a basis for comparison between groups. It is just a list of groups, the result (I think) of a search for labels. He then defies his own effort by declaring himself to be a partial member of a number of different groups -- something he allowed no one else in his writing.

More useful, as Elizabeth Green points out, would be a typology of elements and aspects of beliefs and preferred policies. While I cannot provide a complete inventory or tool, I would like to offer some of the dimensions that might be considered, so we can think about the commonalities and differences between different policies and different policy advocates. (Note: this is off the top of my head, and will be messy, sloppy, incomplete, overlap and fail in any number of ways. Let me know what you think those ways are.)

I. Relationship to Disciplines: Different disciplines take different views, use different tools and concern themselves with different issues. But many disciplines have been brought to education policy. Here are inadequate explanations of the chief relevant disciplines: 
A. Economics: This discipline has been prominent for a number of years. It is very concerned with incentives and markets -- and unlike what you probably think, not nearly as focused on money or dollars. It usually assumes rational actors and has brought us econometrics (e.g. linear regression and other basic statistical tools).
B. Sociology: This discipline has informed education policy for a long time. It looks at society, groups of people, and the impact of social structures, societal forces and norms on individuals. Discrimination and segregation have long been examined through the sociologist's lens, for example. Ethnography and grounded theory come from sociology, though sociologists today use a wide variety of tools.
C. Psychology: Obviously, this discipline is at the root of education. It focuses on how the mind works. In more recent decades, it has shifted some of its focus on how the brain works, and in the last decades, so-called brain-based research has attempted to tell us even more. As I understand it, psychology is more focused on the individual and the nature/mechanism of his/her internal functioning than sociology (which is more interested in the nature of the outside forces and their impact).
D. Law: Obviously, this discipline is about law, regulation and courts. Intentionally crafted rules and precedent -- even when the impacts were not foreseen -- are central. Honestly, I think that this disciplinary approach is rare in education policy, though its subjects matter is often used as a lever.
E. Political Science: This discipline looks at political structures, governance and governmental structures and their functioning. How do decisions get made for a collective or group? There is a focus on relationships between people in different positions and how they influence each other. While some of this looks at intentionally crafted structures, political science also looks at more organic structures and relationships. As in most disciplines, quantitative and statistical tools are quite prominent in the discipline, but not so much for most education policy advocates.
F. Business: I am not sure this is a discipline, but they teach something in business school, right? Management, budgets, finance -- which is not economics, mind you --, logistics. As a method, it relies very strong on case study and best practices, which I believe leads to a pattern of trying to replicate and scale up particular examples. That is, once a good example is found, the goal is to replicate it (as opposed to simply understanding how and why it works, or considering how or why it might not be replicable). Very action focused in its mindset.
G. Education: Even less a discipline than business, but they teach something in ed schools, too. Let's say ideas of teaching and learning, curriculum, classroom organization and techniques. How to work with children. While there is a lot of overlap between these so-called disciplines, education's overlaps with psychology, sociology and business (for education leadership programs) might be the greatest.
H. Common Sense: I'll say it right here. I hate appeals to common sense. I've done a little original research on this term, and I believe that it means "What I already think, and cannot/will not justify with reasoning or evidence." It refuses to engage or critically self-examine.

II. Relationship to Research: There are enormous quantities of education-relevant and even policy-relevant research in each of the disciplines listed above -- and many others. But that does not mean that particular policies or particular advocates all have the same relationship to the research base.  Rather, they may be or do any of the following:
A. Well grounded in the research: Fully aware of the research, both that which supports and that which contradicts preferred positions and/or policies, often requiring including research from a variety of disciplines.
B. Touches on research: Is aware, may cite or build upon research that supports preferred positions or policies, but is not conversant with or aware of contradictory research. Often assumes and declares that the preferred position is notably better supported due to confirmation bias, ignorance, laziness or (at worst) disingenuousness.
C. Coincides with the research: Offers or prefers policies and programs that have support in the research, but has not looked for or been educated in it. Assumes that the research supports their positions, without really looking into it. I am fairly certain that this describes most policy advocates.
D. Coincides against the research: Very much like the prior category, but it happens to be wrong about the research in one of the few areas in which the research is clear.
E. Actively against the research: Is aware that the research argues against their position, but believes that the research is wrong or somehow inadequate for one reason or another.
F. Ahead of the research: Advocates a position in advance of the research having a chance to adequately weigh in. May be proven correct later, but there does not exist a sufficient basis to support claims. Many more advocates believe that they are in this position than actually are.

III. Private Good/Public Good: Some people believe that we should view and judge schools by how well they serve the needs of a student or family, and ask what value the school is giving them. Others see a more public purpose, connected to the common good, improving society and preparing future citizens for their roles as citizens. I think that most people believe that each of these are true, but particular policies often are built on just one or the other. 

IV. The Black Box: No one has such a well developed theory of action that they can explain how every step of everything works. They take some things for granted, acknowledge the limits of their expertise, and/or believe that some things are simply not important.  For example, some advocates of increasing school funding treat tax policy (i.e. exactly how the the money is raised) as a black box. Some who focus on increasing caps on charter schools treat the classroom itself as a black box. Understanding where the black boxes are is key to understanding a policy or position. One way to think about them is based on scope or scale: National Policy -- State Policy/Organization -- District or Network Issue -- School -- Classroom -- Teacher(s) -- Student(s). There are, no doubt, other ways to identify and classify the black boxes.

V. Scale: Do the changes or reforms require national efforts (e.g. at the federal level of government)? Or are they something that individual states can do? Is this a program that an individual teacher can adopt; does it require school level action or even district/network level action?

VI. Time Frame: How long until we will see a pay off? Is this something we expect to see results on in the same year we start? Or is it something that will take much longer to yield results?  

VII. Vantage Point: Is the advocate speaking as a parent? As a teacher? As a local elected official? As a generic business leader? As someone with a product to sell? Is this coming from the ivory tower? Professional advocate? Concerned citizen?

VIII. Experience: How much experience does the advocate have with actual implementations of the programs or policies she/he/it advocates? How much experience does she/he/it have in the context where she/he/it is advocating implementation?

IX. Trust in Standardized Tests: While this is much more closely tied to particular levers and policy ideas, standardized tests and the policies built upon them are at the center of our education policy debate. Understanding someone's stance with regard to standardized tests can help understand why they support the policies they support.
A. High Levels of Trust: Believes that the hard work and professional work that goes into the development of high quality standardized test produce reliable data that give unique and irreplaceable insight into the performance students, teachers and schools. The work of very smart statisticians, psychometricians and others continue to produce data and result that we can rely upon.
B. Medium Levels of Trust: Believes that while standardized test offer valuable data, they provide an incomplete picture and should only be used alongside other -- perhaps more valid -- sources of information when making decisions. 
C. Low Levels of Trust: Believes that while the promise of standardized testing is interesting, the tests we have are flawed enough that they are not useful for wise decision making. Some tests are better than others, but generally they are not ready for serious use in policy or practice -- though hopefully they will be some day (soon?).
D. Absolute Hostility: Believes that standardized tests do not and cannot measure what is important in education. They can only be a waste of time and vehicle for the destruction of meaningful education for students.

X. Preferred Levers: Without having to get down to the level of individual policies, different advocates look to different sorts of levers. I think that this comes the closest to what Mr. Horowitz tried to describe. Of course, individual policies or advocates blend different levers together. (Note that I do not necessarily support or agree with the reasoning behind every lever I list.)
A. Laws: By changing legislation, we can achieve results.
B. Peers: Who a student is with in school makes a difference for that student.
C. Centralization: Bringing control together in one place. 
D. Decentralization: Loosening that central control to give more discretion. Note that this is not just done for the sake of innovation. It can also be done for other reasons, such as the value of variety or the value placed on local (or community) control.
E. Choice: Chubb and Moe famously wrote Choice is a Panacea. This lever includes most "market"-based policies.
F. Funding: Where does the money come from? How much money? How do we decide how much money?
G. Governance: Who runs the schools? Who is responsible for them? Who has control over them? Mayoral control and charter schooling are examples that rely on this lever.
H. Standardization: Applying some common best practices, procedures or tools more widely.
I. Innovation: Perhaps the opposite of standardization. Giving freedom for new ideas and approaches to be discovered and tried.
J. Time: We need to use the time in the school day or year differently, or define it differently.K. Green Sites: Replacing old programs, schools or systems with new ones, usually because the old ones are so dysfunctional that it is easier/better to start over.
L. Leadership: Somehow, addressing just a few people in key positions, we can leverage massive change. (This is where most of my own research falls.)
M. Measurement: We need to have more data on what our schools are producing/accomplishing. 
N. Standards/Curriculum: Focusing on what is taught.  Includes both the formal curriculum and the informal curriculum.
O. Outsiders: There are better ideas, practices and even people outside of our education systems, and if we bring them in then our schools will be much better for it.P. Profit: There is no incentive like money, and utilizing the profit motive can increase performance and innovation -- perhaps even with a local net cost. 
Q. There doubtless are 
R. Rewards: Recognizing good or desired practices or results with rewards.
S. Sanctions: Recognizing bad practices or results with sanction or punishment.T. Teachers & Staffing: Hiring different teachers, training them more/better, retaining/promoting/removing them differently. 
U. Mandates: Simply require action/change.
V. Inducements: Make change or action optional, but try to tip the scales.
W. Capacity Building: Provide/increase ability to do something.  Conservatism: We used to do it better, so let's go back to that. Technology: Modern computer/electronic technology can enable changes that were not previously possible.
X. Widening the Problem Definition: The focus is too narrow, and if we look a bit wider and address the causal forces, we can better solve the problem we are concerned with.Y. Tightening the Ship: The problem is not that we don't know or aren't trying the right things, it's just that we aren't doing it well. We just need to do the same things, but do them the way they are supposed to be done.
Z. Revolution: What we are doing is not even close to what should be done. We need revolutionary and fundamental change.

Again, this is far from complete. And there are a ton of overlaps. But it should give you a framework for examining policies, their advocates and even yourself. If you have any ideas for suggestions or clarification (or to correct mistakes), let me know.

(Edit: Section IX (i.e. standardized tests) added to after original publication.)

How many tests are too many?

How many tests are too many?

How many tests did you take through your school years?

How many chapter tests did you take in your 9th grade math class? How many quizzes? How many spelling tests in second grade? How many pop quizzes in 10th grade science? How long were your midterm and final exams?

I don't know how many class periods I spent just taking a test in high school. I don't know how many times I spent part of the period taking a test. I don't remember how many of the classes had midterms, and how many had formal sit down final exams. I think I know how many AP exams I took, a half day each. I took the SATs twice, a full day each time (I think). I took the PSATs, and I think that was much shorter than the SATs.  My state had POS and SRA tests, mostly before high school. I think the POS tests were annual, but I don't remember how long it took to take them. I think the SRAs were every few years, but I do not remember how often or how long they were. And there were the Achievement tests, too. There were three of those, and they took one hour each.

I am sure that I had to take the "Minimum Competency Test" in 11th grade, and that although my friends and I literally raced through that to see who could finish first, we did not return to our regular classes when we finished. And I do not remember how long was allocated to that test.

If I had grown up in New York, I might have had fewer final exams, but I would probably remember how many Regents exams I took, at a half day, each.

I don't know how much time I spent taking tests, exams and quizzes in school, not even in high school. Was it two full weeks? Was it four weeks? More?

I do remember some tests that I thought were unfair, even ones that I did well on. I remember some teachers loved trick questions. I remember one final exam for math class that everyone thought covered what the class covered, and even people who did poorly on it thought it was a fair test. I also remember a final exam for a math class that had nothing to do with what we covered in class, and no one thought that was fair. I still remember a particular question I got wrong on a standardized test because my AP Chemistry teacher did not think that that one thing was important enough to teach us -- it was about permanganate ions, MnO4-. But other than that, I do not think I ever had a good idea of how well what my teachers taught me matched what they were supposed to teach me.

I don't know the right amount of school time for students to spend taking tests, exams and quizzes. Though I know the strengths and weaknesses of teacher-made assessments, of high quality standardized tests and of low quality standardized tests, I do not know the right ratios of teacher-made tests, locally made tests and commercially created standardized tests.

Do you?

Understanding Teacher Strikes

Teacher strikes are horrendous. They are a challenge for families, who have to scramble for childcare. They can be a nightmare for families who cannot afford to pay for additional childcare. They can be disruptive to students, and life disruptions can seriously impair student learning. They cost teachers paychecks -- at least in the short term. And almost all teachers actually love children  -- else they could not bear to work them with all day every day -- and striking is not nearly as psychically rewarding as teaching. 

So, why do teachers strike? Why don't school boards do what they need to do to prevent strikes?

Well, it is quite complicated, and often needs to be examined on a case-by-case basis. But there are some big historical and/or ongoing issues that you need to understand in order to understand the individual context. 

* Teacher union contracts are not imposed on school districts. They are negotiated between the two parties and agreed to by both parties. 

* The compensation model and teacher tenure predate collective bargaining rights for teachers and union contracts by decades.

* An enormous amount of what is covered in union contracts is a product of neither the union nor the central office trusting principals. This has always been true. Today's decentralizing movement should not hide the fact that districts have historically pushed just as hard as unions to tie principals’ hands and deny them discretion over policy and procedure in their own schools. 

* There has been a problem all across the country with principals failing to observe and evaluate their teachers according to the procedures laid out in the contracts -- even when it takes less than three hours and occurs less often than every year. This failure of evaluation and documentation is what makes it take so long to remove teachers. This does not simply fall on principals, as they are held responsible for so much, are so poorly trained and so poorly supported that it is no surprise that they can have trouble finding time to do this work. And when a new principal comes to a school, s/he may inherit a situation where his/her predecessor failed to observe and evaluate teachers for years. 

(When I work with principals closely, I always ask them if they personally have ever been unable to remove a teacher they needed out of their building. They tell stories about their peers, but I've yet to speak to a principal who admits to being unable to do so him/herself. Not a single one, yet, in scores of conversations.)

* The recent "accountability" movement looks to place blame for student performance on teachers, schools and principals. However, it rarely -- never? -- gives them the discretion over how to accomplish those goals that was originally promised. It certainly fails to provide the capacity that makes those goals even possible -- thus violating Richard Elmore's Principle of Reciprocity. The centralizing tendencies of those who like the idea of "accountability" runs counter to requirements of accountability. For example, if Mayor Rahm Emanuel and the CPS system want to hold principals "accountable" for test scores, as they say, they need to give principals control over their budgets and over staffing. They claim that they that are fighting to give them control over staffing. However, they want to tie the principals' hands when it comes to evaluating their own teachers.

* Most individual teachers get raises above and beyond those highlighted in media coverage of teacher contract disputes. They get raises for advancing in their careers, as defined in the salary schedules in the contracts. Whether you agree or disagree with that approach, there are raises in there. The public disputes are about shifting that entire salary schedule -- to adjust for inflation, to make the profession more attractive, and to pay teachers closer to that a professional salary. If you believe that teachers are underpaid or that starting teacher salaries need to be higher -- as most do -- then you should be in favor of these overall shifts. On the other hand, teachers should not claim that their individual raises depend entirely upon these shifts.

* We all know that teaching is hard, hard work. Working in front of dozens of children everyday, being minutely observed by them and trying to act like a role model every minute is incredibly draining. After classes are over, teachers have papers to grade, lesson plans to write or revise, meetings to attend, phone calls to make (e.g. to parents), classrooms to clean and the next day's lessons/activities to set up. Teachers work long after the children leave school.

* Teachers do not have three months off every year. They generally have 10-month contracts, yes. Some teachers -- especially well-experienced teachers -- do not have to spend their summers planning curricula, learning more about their subjects, attending training and professional development. But many teachers do. Furthermore, many, many teachers (most?) try to get back to school before they are supposed to, to begin setting up their classrooms --which may have moved -- or prepare for their new classes and grade-levels. Most teachers need multiple weeks before the students arrive at the start of the year to get ready, and they are only paid for one.

* All professionals need insightful feedback to build on their strengths and to address their weakness. And yet, it is an entirely human reaction to resist criticism. Amid the traditional close your door and do your own thing culture of schooling, there can be even more resistance to evaluation. (To be honest, I was quite resistant to the obviously stupid criticism I received as a teacher.) But there are legitimate reasons to want to compare teacher performance, even across schools (e.g. targeting professional development, identifying model teachers, etc.).

* Student test scores do not measure school quality. However, there are efforts to use statistical techniques to combine test scores and other data to get closer to teachers and school quality. Yet these products -- called "value-added analysis" (VAA) -- are not even close to being ready for primetime. To take two disparate examples, VAA does not account for the presence of known disruptive students or for HVAC issues in a particular classroom -- both of which can significantly impact student learning. Until far more of the known relevant factors are accounted for, VAA systems simply do not achieve their goal. (Though, given valid tests that appropriately sample from the content domain, fuller models, and sufficient data, VAA can be an incredibly useful tool.)

* Despite the rhetoric, the United States has never led in the international comparisons of student performance. PISA and TIMMS studies have consistently shown that US to be in the middle of the pack of industrialized nations -- for their entire histories. More recent research has show that when you control poverty levels, however, the US is today first in the world on these tests. We have not declined, but we have social problems that impact student performance. 

Many (most? all? nearly all? including myself) believe that our schools need to do better. Certainly, we want our students to come out of school better prepared for citizenship, their future careers and their lives as adults. Many parents need to learn how to better support their children's' education. Societally, we need to value education and scholarship more. We need better teachers and better teaching. We need better leadership of our schools. We need better leadership of our school districts. The visions have been lacking. The support for those in schools has been poor. The necessities of creating engines of continuous improvement have not been met -- either for students or for educators. 

When reading or thinking about the Chicago Teachers Union strike, keep all of this in mind. The next time you hear about a teacher contract negotiation, keep all of this in mind. And if you care to know more any any of these points, there is a ton of good work out there to learn from.

Putting Children First

Obviously, when it comes to education policy, we need to put children first. That is why when we are faced with the potential of teacher layoffs, we need to make sure that we abolish seniority as a criterion. Because we value children and education so much, we must make sure that the least “effective” teachers are laid off first, not the least senior. After all, children first.

Or so say an enormous number of people, relatively few of whom actually work in classrooms every day.

Most of the arguments against this line of argument address the question of teacher effectiveness and how we might recognize it. And there are a lot of good objections there. But I want to address the ridiculous idea of putting the needs of students ahead of the needs of adults. I want to question this idea of children first.

Over at Gotham Schools, I challenged commenters to come up with a workable alternative to seniority for layoff decision-making. That meant an alternative that would cover all teachers, would not be subject to easy manipulation (e.g. from favoritism or backlash), and would produce to unambiguous priorities in layoffs. The only response was a Bad Attendance First Out (BAFO) plan.

As I wrote over there, BAFO is legally unworkable. It could run afoul of the federal Americans with Disabilities Act Family and Medical Leave Act, the federal Uniformed Servicemembers Employment and Re-Employment Rights Act and the New York Human Rights Law. That's a problem. But in my view, that's not the biggest problem. The biggest problem is the children first thinking behind this proposal.

Should teachers be allowed to take sick days? Should they be allowed to take their own children to doctor's appointments? Should they be allowed take a day off to attend a funeral? What about their parent's funeral? What about -- god forbid -- their own child's funeral? Or, should they suffer the punishment of being higher on the layoff list if they do any of these things?

What about being stuck on a train that has had mechanical problems? What about their car being broadsided by a car that ran a red light? What about having a heart attack and therefore being hospitalized? Should these teachers  be held responsible for the time they miss and be put higher on the layoff list?

If you want to suggest that teachers get too many vacation days in addition to the scheduled school vacations, I might agree with you. But sometimes, teachers need personal days. Sometimes they really need sick days.

The children first thinking of so many non-teachers is premised on the idea that teachers should have less rights than any other workers in this country, because -- after all -- children first. Unlike other workers, they should not be able to take vacation. Unlike others, they should not be allowed to get sick. Unlike others, they should not be allowed to go to funerals. Unlike others, nothing in their own personal lives should ever be more important than their jobs. They should take pay cuts, give up the pensions they were promised, suffer arbitrary (and often capricious) judgments at the hands of their undertrained and undersupported supervisors. Unlike others, they should give up their lunch hours and work beyond any reasonable number of “work hours.”

According to non-teachers, the laws that protect all other workers and the rights of all other workers should not apply to teachers. The contracts that schools and districts sign with teachers should not be binding on the schools and districts. The standards for workplaces and treatment of employees that matter in other contexts should not apply to schools and teachers. After all, children first.

I'm sorry, but we cannot always put children first.

Oh, wait......I'm not sorry.