Thursday, March 7, 2013

In Search of a Better Education Policy Advocate Taxonomy

On March 5, Eric Horowitz posted In Search of a Better Education Policy Taxonomy on his blog, Peer-reviewed by my neurons. However, he does not offer a taxonomy of education policy. Rather, he posts a taxonomy of education policy (i.e. "reform") advocates. What he really does is try to put advocates into different groups, but without offering a basis for comparison between groups. It is just a list of groups, the result (I think) of a search for labels. He then defies his own effort by declaring himself to be a partial member of a number of different groups -- something he allowed no one else in his writing.

More useful, as Elizabeth Green points out, would be a typology of elements and aspects of beliefs and preferred policies. While I cannot provide a complete inventory or tool, I would like to offer some of the dimensions that might be considered, so we can think about the commonalities and differences between different policies and different policy advocates. (Note: this is off the top of my head, and will be messy, sloppy, incomplete, overlap and fail in any number of ways. Let me know what you think those ways are.)

I. Relationship to Disciplines: Different disciplines take different views, use different tools and concern themselves with different issues. But many disciplines have been brought to education policy. Here are inadequate explanations of the chief relevant disciplines: 
A. Economics: This discipline has been prominent for a number of years. It is very concerned with incentives and markets -- and unlike what you probably think, not nearly as focused on money or dollars. It usually assumes rational actors and has brought us econometrics (e.g. linear regression and other basic statistical tools).
B. Sociology: This discipline has informed education policy for a long time. It looks at society, groups of people, and the impact of social structures, societal forces and norms on individuals. Discrimination and segregation have long been examined through the sociologist's lens, for example. Ethnography and grounded theory come from sociology, though sociologists today use a wide variety of tools.
C. Psychology: Obviously, this discipline is at the root of education. It focuses on how the mind works. In more recent decades, it has shifted some of its focus on how the brain works, and in the last decades, so-called brain-based research has attempted to tell us even more. As I understand it, psychology is more focused on the individual and the nature/mechanism of his/her internal functioning than sociology (which is more interested in the nature of the outside forces and their impact).
D. Law: Obviously, this discipline is about law, regulation and courts. Intentionally crafted rules and precedent -- even when the impacts were not foreseen -- are central. Honestly, I think that this disciplinary approach is rare in education policy, though its subjects matter is often used as a lever.
E. Political Science: This discipline looks at political structures, governance and governmental structures and their functioning. How do decisions get made for a collective or group? There is a focus on relationships between people in different positions and how they influence each other. While some of this looks at intentionally crafted structures, political science also looks at more organic structures and relationships. As in most disciplines, quantitative and statistical tools are quite prominent in the discipline, but not so much for most education policy advocates.
F. Business: I am not sure this is a discipline, but they teach something in business school, right? Management, budgets, finance -- which is not economics, mind you --, logistics. As a method, it relies very strong on case study and best practices, which I believe leads to a pattern of trying to replicate and scale up particular examples. That is, once a good example is found, the goal is to replicate it (as opposed to simply understanding how and why it works, or considering how or why it might not be replicable). Very action focused in its mindset.
G. Education: Even less a discipline than business, but they teach something in ed schools, too. Let's say ideas of teaching and learning, curriculum, classroom organization and techniques. How to work with children. While there is a lot of overlap between these so-called disciplines, education's overlaps with psychology, sociology and business (for education leadership programs) might be the greatest.
H. Common Sense: I'll say it right here. I hate appeals to common sense. I've done a little original research on this term, and I believe that it means "What I already think, and cannot/will not justify with reasoning or evidence." It refuses to engage or critically self-examine.

II. Relationship to Research: There are enormous quantities of education-relevant and even policy-relevant research in each of the disciplines listed above -- and many others. But that does not mean that particular policies or particular advocates all have the same relationship to the research base.  Rather, they may be or do any of the following:
A. Well grounded in the research: Fully aware of the research, both that which supports and that which contradicts preferred positions and/or policies, often requiring including research from a variety of disciplines.
B. Touches on research: Is aware, may cite or build upon research that supports preferred positions or policies, but is not conversant with or aware of contradictory research. Often assumes and declares that the preferred position is notably better supported due to confirmation bias, ignorance, laziness or (at worst) disingenuousness.
C. Coincides with the research: Offers or prefers policies and programs that have support in the research, but has not looked for or been educated in it. Assumes that the research supports their positions, without really looking into it. I am fairly certain that this describes most policy advocates.
D. Coincides against the research: Very much like the prior category, but it happens to be wrong about the research in one of the few areas in which the research is clear.
E. Actively against the research: Is aware that the research argues against their position, but believes that the research is wrong or somehow inadequate for one reason or another.
F. Ahead of the research: Advocates a position in advance of the research having a chance to adequately weigh in. May be proven correct later, but there does not exist a sufficient basis to support claims. Many more advocates believe that they are in this position than actually are.

III. Private Good/Public Good: Some people believe that we should view and judge schools by how well they serve the needs of a student or family, and ask what value the school is giving them. Others see a more public purpose, connected to the common good, improving society and preparing future citizens for their roles as citizens. I think that most people believe that each of these are true, but particular policies often are built on just one or the other. 

IV. The Black Box: No one has such a well developed theory of action that they can explain how every step of everything works. They take some things for granted, acknowledge the limits of their expertise, and/or believe that some things are simply not important.  For example, some advocates of increasing school funding treat tax policy (i.e. exactly how the the money is raised) as a black box. Some who focus on increasing caps on charter schools treat the classroom itself as a black box. Understanding where the black boxes are is key to understanding a policy or position. One way to think about them is based on scope or scale: National Policy -- State Policy/Organization -- District or Network Issue -- School -- Classroom -- Teacher(s) -- Student(s). There are, no doubt, other ways to identify and classify the black boxes.

V. Scale: Do the changes or reforms require national efforts (e.g. at the federal level of government)? Or are they something that individual states can do? Is this a program that an individual teacher can adopt; does it require school level action or even district/network level action?

VI. Time Frame: How long until we will see a pay off? Is this something we expect to see results on in the same year we start? Or is it something that will take much longer to yield results?  

VII. Vantage Point: Is the advocate speaking as a parent? As a teacher? As a local elected official? As a generic business leader? As someone with a product to sell? Is this coming from the ivory tower? Professional advocate? Concerned citizen?

VIII. Experience: How much experience does the advocate have with actual implementations of the programs or policies she/he/it advocates? How much experience does she/he/it have in the context where she/he/it is advocating implementation?

IX. Trust in Standardized Tests: While this is much more closely tied to particular levers and policy ideas, standardized tests and the policies built upon them are at the center of our education policy debate. Understanding someone's stance with regard to standardized tests can help understand why they support the policies they support.
A. High Levels of Trust: Believes that the hard work and professional work that goes into the development of high quality standardized test produce reliable data that give unique and irreplaceable insight into the performance students, teachers and schools. The work of very smart statisticians, psychometricians and others continue to produce data and result that we can rely upon.
B. Medium Levels of Trust: Believes that while standardized test offer valuable data, they provide an incomplete picture and should only be used alongside other -- perhaps more valid -- sources of information when making decisions. 
C. Low Levels of Trust: Believes that while the promise of standardized testing is interesting, the tests we have are flawed enough that they are not useful for wise decision making. Some tests are better than others, but generally they are not ready for serious use in policy or practice -- though hopefully they will be some day (soon?).
D. Absolute Hostility: Believes that standardized tests do not and cannot measure what is important in education. They can only be a waste of time and vehicle for the destruction of meaningful education for students.

X. Preferred Levers: Without having to get down to the level of individual policies, different advocates look to different sorts of levers. I think that this comes the closest to what Mr. Horowitz tried to describe. Of course, individual policies or advocates blend different levers together. (Note that I do not necessarily support or agree with the reasoning behind every lever I list.)
A. Laws: By changing legislation, we can achieve results.
B. Peers: Who a student is with in school makes a difference for that student.
C. Centralization: Bringing control together in one place. 
D. Decentralization: Loosening that central control to give more discretion. Note that this is not just done for the sake of innovation. It can also be done for other reasons, such as the value of variety or the value placed on local (or community) control.
E. Choice: Chubb and Moe famously wrote Choice is a Panacea. This lever includes most "market"-based policies.
F. Funding: Where does the money come from? How much money? How do we decide how much money?
G. Governance: Who runs the schools? Who is responsible for them? Who has control over them? Mayoral control and charter schooling are examples that rely on this lever.
H. Standardization: Applying some common best practices, procedures or tools more widely.
I. Innovation: Perhaps the opposite of standardization. Giving freedom for new ideas and approaches to be discovered and tried.
J. Time: We need to use the time in the school day or year differently, or define it differently.K. Green Sites: Replacing old programs, schools or systems with new ones, usually because the old ones are so dysfunctional that it is easier/better to start over.
L. Leadership: Somehow, addressing just a few people in key positions, we can leverage massive change. (This is where most of my own research falls.)
M. Measurement: We need to have more data on what our schools are producing/accomplishing. 
N. Standards/Curriculum: Focusing on what is taught.  Includes both the formal curriculum and the informal curriculum.
O. Outsiders: There are better ideas, practices and even people outside of our education systems, and if we bring them in then our schools will be much better for it.P. Profit: There is no incentive like money, and utilizing the profit motive can increase performance and innovation -- perhaps even with a local net cost. 
Q. There doubtless are 
R. Rewards: Recognizing good or desired practices or results with rewards.
S. Sanctions: Recognizing bad practices or results with sanction or punishment.T. Teachers & Staffing: Hiring different teachers, training them more/better, retaining/promoting/removing them differently. 
U. Mandates: Simply require action/change.
V. Inducements: Make change or action optional, but try to tip the scales.
W. Capacity Building: Provide/increase ability to do something.  Conservatism: We used to do it better, so let's go back to that. Technology: Modern computer/electronic technology can enable changes that were not previously possible.
X. Widening the Problem Definition: The focus is too narrow, and if we look a bit wider and address the causal forces, we can better solve the problem we are concerned with.Y. Tightening the Ship: The problem is not that we don't know or aren't trying the right things, it's just that we aren't doing it well. We just need to do the same things, but do them the way they are supposed to be done.
Z. Revolution: What we are doing is not even close to what should be done. We need revolutionary and fundamental change.

Again, this is far from complete. And there are a ton of overlaps. But it should give you a framework for examining policies, their advocates and even yourself. If you have any ideas for suggestions or clarification (or to correct mistakes), let me know.

(Edit: Section IX (i.e. standardized tests) added to after original publication.)


  1. Ceolaf,

    Here's my first take.

    Obviously, your post is better than mine. I have long been wrestling with this, however, and I'll incorporate your thoughts.

    Regarding Economics, I'd add two things. It is a shame that the discipline of Political Economy has virtually disappeared. And, with so much trust in modeling, they seem to think the ability to control for factors has replaced the need for qualitative analysis at each step of the way with quantitative analysis.

    I'm glad you cited Trust in Standardized tests. We also need Truast in Relational Trust. How comfortable do different groups feel with, say, Deasy, signing a CBA and then ordering his principals to disobey it? How comfortable are different groups with "bait and switch" tactics, with scorched earth polititics? How comfortable are they with "papers" that incorporate sophisticated statistics but start with outright political spin, mischaracterize their opponents, and hide the key parts of the methodology?

    I could go on. You stimulated so many good thoughts to wrestle with.

    However, I trust Common Snse more than most metrics. My position is that you improve schools the way you improve a football team. You reduce unforced errors.

  2. I posted a quick update on my post that mentions this, but one issue that really drives the charter school debate is equalizing opportunities among the entire population of students vs. equalizing opportunities among subsets of students. For example, I think you can argue that a CMO like KIPP equalizes opportunities between low-income and wealthy students while at the same time doing the opposite among subgroups of low-income students.