There's no use inveighing on education policy until after season 4 of The Wire. We haven't yet established all the issues.
or a repeal? I'd like to repeal all the fake-sounding laws. Maybe that asks too much.
So, are grassroots Democrats going to get all pissy when our personal pet issues aren't addressed in a timely manner? Not that this is a precursor to that. But I hope we don't fall into that trap.
Organizations can too respond to incentives. The problem with NCLB is that it has stupid metrics and stupid incentives, not that the idea of incentives for institutions is confused.
No, the decision-makers in organizations can respond to incentives that affect them. If the interests of the decision-maker are closely enough aligned with the interests of the organization, it can look as if the organization is responding to an incentive. But organizations have no minds, no hopes, no fears, and no capacity to take action on their own behalf -- they operate only through people, who respond to incentives.
Is describing this fact as 'the agency problem' a lawyer thing, or is that a generally familiar term?
Organizations can respond to incentives. The school isn't greater than the some of its parts. It has a superintendent. It has a parents' board. It is funded by local taxes. These groups can react to public pressure. If they are the competent sort that reacts, you bet they'll be motivated by finding out that their school is labelled a failure.
To think otherwise is to argue that Congress can't be moved to do anything, because a Congress doesn't care what happens to it, only elected officials do.
The problem is that the metrics in NCLB are dumb. Testing doesn't show a damn thing, unfortunately, except that the students are being taught to the test.
That said, I'm not sure that incentives will be the best method to fix the schools, as the incentive structure seems to presuppose that the problem with bad schools is lazy teachers.
It's not a lawyer thing. It's just that the 'school' caring here is just a shorthand way of saying 'the principals, administrators, PTA boards, and teachers', and those guys can too be affected by incentives. It's just that these incentives and metrics suck.
Right, the problem is that the incentives suck. But I think they suck because they're conceived of as punitive toward the school, as if it cared, rather than as incentivizing toward the people who run the school. (Oh, you could aim punitive incentives toward the people as well -- fire the principals and all the teachers in underperforming schools? But people wouldn't go for it, because it's silly, and where would we get replacements from?)
Oh, Cala and I are gonna tag-team LB with TEH PHILOSOPHY OF ACTION. If anyone is turned on by this, seek help.
The institution's response to incentives will supervene on individual agents and their incentives, but it is still true to say that the school is responding to incentives because that just is a matter of having certain individuals acting in certain ways. (To say "no, it's just these individuals, not the institution" is to make a category mistake.)
Compare to ascriptions of mental states to groups: IBM is worried about...while Microsoft believes that...etc.
I've never understood why such stigma is attached to the phrase, "being taught to the test."
Admittedly, if you have poor tests, then teaching to them is sub-optimal.
But certainly, there exist some tests for which the classes that "teach to them" are in fact covering the appropriate material, at an appropriate speed, with an appropriate emphasis on "understanding" such that the content of the class can be successfully recalled in a testing situation. Then the test becomes a way of simultaneously adjusting all the courses/classes/teachers within its purview -- a driver's wheel for the entire school system. Not necessarily a bad thing, no?
Also, the idea of transferring teachers from high-performing schools to low-performing schools gives me the heebie-jeebies, at least a bit. If you were the principal of a "good" school, and you liked working with your team of teachers, wouldn't this give you a disincentive, a motivation to make sure your teachers aren't "too good?" If you are a good teacher who likes the school he/she teaches in, wouldn't this discourage you from trying to completely optimize your students' performance (on the test)?
Okay, can I describe the error as assuming that organizations respond to incentives in a manner that has a simple relationship to the way people respond to incentives? Just because a person doesn't want to lose limbs, for example, and will try to avoid that happening, doesn't mean that an organization doesn't want to lose members. Unless the people making the decisions feel a personal negative effect, the organization will not spontaneously avoid 'bad things' happening to it.
So if you're trying to make an organization change its behavior, treating it as if it were an organism will lead you astray. You have to specificially address your incentives to the decision-makers.
But I have to run to a meeting now -- I'll continue this later.
Of course we might correctly ascribe beliefs to Microsoft based in its actions (er, for some noncircular way of ascribing actions to Microsoft) that actually no one individual at Microsoft holds.
11 seems right, but is also consistent with saying "the school responds" or "the committee responds", but is totally not consistent with saying the 'some' of all parts, for which I now chastise myself.
10: I think that test only exists on unicorn island. Even good comprehensive university exams, which are difficult to construct, suffer from the problem of having students try to cram for the exam rather than engage and absorb the material. Also, music, gym, art &c isn't tested, so we get a bunch of kids crying through their algebra homework while pumped up on Ritalin because for some reason they have so much energy.
Pelosi is just Speaker not Goddess of the Known World. She will be pretty busy, and it might be a good idea to learn the names and contact numbers of the Committee Chairpersons and congresspeoples. Like for instance, the ones on the Education Committee and subcommittees.
Just sayin.
music, gym, art &c isn't tested, so we get a bunch of kids crying through their algebra homework while pumped up on Ritalin because for some reason they have so much energy
This is brilliant.
14: With the caveat that no test will ever be perfect in the way that I describe, and with the additional understanding that some things should not be tested (the example of testing kindergartner's from teo's blog is, very obviously, ridiculous) ... I respectfully disagree that an effective test is solely within the realm of "unicorn land."
I would argue that a test could fail along roughly two lines: it is either too narrow in scope, or its content doesn't accurately reflect the subject material itself. In the first case, "teaching to the test" (TttT) is harmful because it gives an incentive to teachers to leave out useful/interesting but non-covered material. In the second case, TttT devolves into simply teaching techniques or rote-memorization for the test: memorizing vocabulary, practicing fixed forms of logic problems, etc. Any test for a subject like "art" or "music" is likely going to fall into the first category, and (I'd say) tests like the SAT and GRE are in the second.
But that doesn't mean that effective tests don't exist -- I can only speak from personal experience and anecdote, but I would argue that a significant number of the high-school AP and IB exams do a pretty decent job at hitting the twin-notes of breadth and accuracy. The writing components help their cause, really.
Obviously, there are problems of scale involved in those tests, which would make 'scaling them up' to a national level more difficult, but... does someone really want to argue that studying the content of the AP Calculus exam isn't a pretty comprehensive review-and-test of simple, single-variable calculus?
Gym and music/art seem like special cases, and I don't think should affect my point about tests in general. Just b/c we (might) want to have a comprehensive system of testing in our schools doesn't mean that only testable material is to be taught in all classes, after all.
In practice, that's the result. If my school is in danger of being marked failing in mathematics, and I have X dollars to spend to correct that, I'm going to drop the music curriculum in favor of more math classes.
Part of my resistance stems from the fact that I found the AP tests, like all standardized tests, to be very easy to figure out. You can eliminate answer choices based on how politically correct they are. The theme of the story is never going to be that 'Racial minorities are naturally less gifted and therefore we should discriminate against them.' The writing section helps, and I think you can test math and science well with exams, but literature?
I've never understood why such stigma is attached to the phrase, "being taught to the test."
It's used as a slur because the behavior it describes is worthy of it. Kids memorizing answers without understanding WHY the answer is what it is aren't learning. They're just temporarily storing information. There's 40+ years of research on why memorization hurts learning.
Last week I tried in vain to gently walk a colleague through an explanation of why it was not possible that having served 9 out of 36 students meant that she reached 3%. I'm sure she did quite well at memorizing formulas in school, but she hasn't developed an ability to understand why 3% is a ridiculous answer to come up with. Never mind that she applied the wrong formula in calculating the percentage -- she didn't have the baloney-detector or the self-confidence to say "Wait a minute! That can't possibly be right."
As far as good tests - yes, they're rare. In part because they can't be machine-graded. If I test your communications skills by rating how fast you can dial a 1-800 number and how many URLs you have memorized, and you get a high score, I still don't know much. If I ask you to track down a person in another state, call them, get information and make a decision based on it, I'll know a lot more.
Schools in which there are learning problems should get more resources, not less. The whole punitive aspect of that bill makes absolutely no sense if it is truly supposed to make sure that no child is left behind. (the ways they give these counterproductive bills positive names are just completely over the top.)
As for institutional incentives, as many of you pointed out the incentives in this bill don't necessarily actually result in better organizational behavior on the parts of affected schools. Also test results may point to one problem or to another. How do you know which problem the school has? And isn't it just too easy to blame some discrete function (teachers, admin, whatever) without really knowing what is causing the problem?
Not to mention that, as Bob Somerby (the daily howler, he often posts on education issues) has said, if a school already has a lot of kids who are "behind grade level" and is given even more pressure to get them up to speed and pushes the kids who are already behind it is not going to necessarily succeed and the kids will be even more demoralized. He however does not think this problem is very easy to define or to solve and that a large part of the problem is people oversimplifying it without even trying to understand education data etc. (especially the media).
Well, maybe I would settle for a rigorous national program of testing in the maths and sciences (and maybe some history) then.
As for this:
"If my school is in danger of being marked failing in mathematics, and I have X dollars to spend to correct that, I'm going to drop the music curriculum in favor of more math classes."
... maybe all I can say is, you'd obviously need some kind of compensatory requirements for subjects that can't easily be tested. If I had absolute control over the American educational system, any requirement that I imposed as to math and science testing would be accompanied by some additional restriction: "maximize your test scores, subject to the constraint that all students still receive X hours of weekly instruction in art, music, gym, etc by teachers who've been certified in their instruction," or something along those lines.
Like a linear programming problem, but with schools.
Part of the problem of school reform is is that some students have greater aptitude than others. Asians and Jews do better in schools than non- jewish whites who do better in schools than blacks. Not uniformly but in a non-trivial way and it isn't just a proxy for economic class.
This is typically ignored education policy, but it screws up the incentives. Depending on the student population mix, some high value added schools can fail the NCLB tests and other low value added schools can easily pass. I have no suggestions on how to fix things.
...though with the regime change about to happen for the 110th, I'm sure a lot of this is up in the air.
http://edworkforce.house.gov/members/109th/mem-fc.htm
Part of the problem of school reform is is that some students have greater aptitude than others. Asians and Jews do better in schools than non- jewish whites who do better in schools than blacks. Not uniformly but in a non-trivial way and it isn't just a proxy for economic class.
In this context, "aptitude" gets it exactly wrong.
Awesome.
Cala: The theme of the story is never going to be that 'Racial minorities are naturally less gifted
Some random dude: some students have greater aptitude than others. Asians and Jews do better in schools than non- jewish whites who do better in schools than blacks.
I looked at the incentives for teachers and schools in the NC public school system a while back. It turns out that the optimal strategy for maximizing financial gain would be to take a dive one year, then gradually improve a little bit every year for five or six years, whereupon you'd take a dive again.
I've never understood why such stigma is attached to the phrase, "being taught to the test."
My niece was a dinosaur buff in primary school and asked how to spell "stegasaurus". The teacher wouldn't tell her because it wasn't on the test.
Chinese and French school systems do teach to a national test, and the results aren't terribly bad. To a certain extent it means that everyone in the country is competing with everyone else. In my experience Chinese students "study to the test" and think of study as cramming.
Teaching to the test is bad when the test is all there is, and when teaching to the test is all that is done.
The current regime of high-stakes testing serves students (and teachers) very poorly, and the disincentives attached to poor test scores unjustly penalize the schools that need more money, not less.
I just read Jonathan Kozol's Shame of the Nation, so I am currently brimming with fizzy outrage.
Thanks for the link, LB. A lot of the comments here are great.
28: Okay, okay... Teaching to the test is bad when the teacher knows exactly what's on the test. Then the test can be aced by simply memorizing what's on it, teaching to it would be very harmful, and so forth. Even if you don't know the exact content of the test, you can still game some exams (witness Kaplan classes for the SAT or GRE, or things like that).
In other places, this would be called "overfitting," and it's not-a-good-thing.
But contrast this with the way that (I claim!) the AP exam works, for something like calculus: the core document is the published collection of subjects/topics which will be covered, along with an additional collection of topics which may be covered. The form and number of questions is outlined, times are given, etc. The point being: teaching to the part of the test about definite integrals (for instance) is best accomplished by teaching the students what a definite integral is and how to use it. It's math (hence: symbolic manipulation), so a certain amount of memorization and practice will be required -- those aren't necessarily bad things.
All I'm saying is, you can't condemn educational systems which depend (in part) on regular testing simply by claiming, "it's bad when teachers teach to the test." Well-designed tests aren't simply Unicorn Fantasies, and teaching to them can be a good thing.
I'm sure I'm beating a dead horse, so I'll stop ...
You're right, arthegall, but the problem is that you can't teach all subjects like that. In fact, you really can't even teach most subjects like that; only math and the hard sciences, of subjects taught at K-12 level, are really clear-cut enough conceptually that rigorous testing is valuable. For the other subjects, high-stakes testing just isn't the best way to tell if kids are learning the material, so teaching it in preparation for a test is totally counterproductive.
Once upon a time I was very nearly a middle school math teacher, so I'll throw that caveat out before I play devil's advocate: if the system as we have it punishes schools that do poorly by removing resources the scarcity of which is already indicated by the poor performance, doesn't reversing it so that schools that do poorly get more resources simply incentivize them to do poorly so they'll get more money?
I don't actually believe this, or think NCLB was a good idea, or like standardized tests at all (TttT is one of the reasons I was nearly an educator). I do think it's worth considering, however, that any system which includes (dis)incentives tied to performance can be gamed and in a field as desperate for resources as education generally is that system will be gamed. Rather than tie rewards or punishments to artificial benchmarks - rather like standardized testing itself, isn't it? - a better approach would be to create different metrics entirely. I'm with arthegall in that I think standardized tests are a pretty damn fine way to test math and the hard sciences and the rote memorization portion of pretty much any subject (standardized tests would fit music theory just as well as they do math), and have zero objection to setting good standards for those. At the same time, though, I'd just as soon scrap standardized testing for, say, English Lit and instead fund better-stocked libraries and require a certain percentage of class time be devoted to reading or a certain number of book reports or something. For that matter, I'd be fine with chopping the science tests in half for most science subjects and instead providing the resources and requirement that half of all class time be spent doing lab work.
Teaching to the test is bad because it conveys to students--even college students--that the point of learning is to pass, rather than to actually *learn*. Testing is a fine way of assessing basic knowledge; it's a very bad way of assessing thinking or the ability to apply what you know. It has its place, but so do subjective assessments and longer-term projects.
The other problem with testing is that it's a highly artificial situation that doesn't have much applicability outside of education. A lot of students develop a lot of anxiety around testing, which has been shown to seriously impair their performance. Those same students, given different kinds of assignments that allow them to think, revise, or approach problems in their own way, often demonstrate a lot better understanding of the subject than they do on tests.
So part of what tests assess is basic knowledge; another part is testing skill. The former's useful; the second, eh.
FWIW, I actually think that you could probably include history ("social studies," blah), foreign language instruction, and some "soft" sciences (why only restrict to the hard stuff?) to the list of subjects which could be fruitfully tested, with enough resources.
Clearly the total list of those subjects shouldn't exhaust the set of things that should be taught in schools -- and I'll admit, I'm thinking mainly of high-school, here.
And I'll be the first to vote for "more resources" for schools, too. Testing should clearly be tied to increased funding.
But even in English classes, better resources (bigger libraries?) are only part of the story. I'm a product of the NC public school system, and I feel like I'm a smart guy, but I never would have gone to the library and checked out Madame Bovary or Invisible Man if there hadn't been a test scheduled for both those books...
And even though I crammed for that test, and I'm sure I've now forgotten most of what I had memorized then, it did get me to read them, and I am happy I did so...
FWIW, I actually think that you could probably include history ("social studies," blah), foreign language instruction, and some "soft" sciences (why only restrict to the hard stuff?) to the list of subjects which could be fruitfully tested, with enough resources.
See, I think most of this stuff would be much better taught with papers and presentations rather than tests. Not that there shouldn't be any tests at all, just that they shouldn't be the focus the way they should be for math and science. For history, e.g., there just isn't the discrete body of topics that have to be covered that there is in calculus, so even a well-designed test won't necessarily be a good mechanism for judging whether the kids really "get it." The same for the other subjects. Testing has its place, and I'm not saying it should be dispensed with entirely, but there's very little chance of that happening anyway so I think it's better to focus on reducing the (insane) amount of testing we already do.
I'm a product of the NC public school system, and I feel like I'm a smart guy, but I never would have gone to the library and checked out Madame Bovary or Invisible Man if there hadn't been a test scheduled for both those books...
I'm also a product of the NC public school system, and although we were a very book-obsessed family and so I would have turned out a reader anyway, there are definitely books I never would have read without a class assignment but that I was later glad had been forced upon me (Wuthering Heights, which I never would have chosen on my own, is a personal fave). I think that's a pretty valid way to describe the reading history of just about every student, and so I'd rather see the metric be "the student must read and in some way be measured on comprehension of X books, or as Y% of the total curriculum" than anything else. My goal would be to create a system in which the metric is not a percentage of correct answers on a test (though I agree that testing can be applied to a lot of sciences other than, say, physics) but a quantity and quality of range of experience. I have no idea how that metric would be written or assessed, but that would be how it worked for me in fairy-unicorn land's only slightly more practical neighboring nation.
I've taught in a 'teach to the test' environment (Samoa, which has a British-kinda terminal exams in secondary school system), and I'm on both sides. If your students are at pretty close to the level the test expects them to be, and the actual subject you're trying to teach fits well into a test format, it can be good -- I think we'd be better off with a system where highschool kids nationally all had to take the same rigorous you-must-know-these-three-hundred-historical-facts exam. (Yes, I know memorization is bullshit, but it's a lot easier to have deep thoughts about history if you also have a strong sense of, you know, what happened, when.)
The problem with NCLB is that what it's really about is reading (I don't think the math exams are nearly as much of a problem). And taking a standardized exam, while it's something that good readers are going to do well, is a very different skill from being a fluent reader, so teachers have a choice between teaching their students to be better readers, or to be better takers of exams meant to test reading. Where the skills needed to pass the exam don't match up well to the real-world-skill you want to teach, teaching to the test is a very bad thing.
Where the skills needed to pass the exam don't match up well to the real-world-skill you want to teach, teaching to the test is a very bad thing.
Right, exactly. Teaching to the test can be fine in a subject where doing well on the test is a good proxy for knowing the material, and there are subjects like that (math in particular), but it becomes a real problem when understanding the subject just happens to correlate with doing well on the test without there necessarily being any causal connection (as in reading). And of course when the students start out pretty far behind the level of the test (as is often the case with NCLB) there's a whole bunch of other problems for the teacher.
Right. I was in that position in Samoa, teaching a math class to a test that was sort of halfway between Calc and pre-Calc -- derivatives, easy integrals, but probably not quite what's on the calc AP. My students could (mostly) hardly do basic arithmetic, and had a hell of a time with any algebra at all. The right thing to do would have been to drop the class back to where they were, and teach them algebra with some remedial arithmetic. School policy was to aim for the test, so I taught some cookbook formulae (the power rule is pretty easy), and got them higher grades (failing, but higher) on the exam than I would have if I'd taught them more math.
I'm sure long-time unfoggeditarians know this, LB, but a) how did you end up in Samoa b) teaching math?
If a bunch of business people sincerely tried to make schools better, I suspect that they would [did?] come up with something like NCLB.
They'd reason like this: The hot breath of competition makes busineses excel, so we'll get excellent schools by forcing them to compete. Winners will get more money and expand, while losers will get starved and shut down. Eventually, only the winners will remain. Right then, so we need clear (read numeric) metrics; and real (read financial) rewards and penalties.
If that sounds like a cartoon, try discussing the subject sometime with a room full of MBAs. And remeber that NCLB is a the product of our first MBA president. It's how they think.
For fun, point out to them that America's schools lag France's or Japan's on some measure or other. Aha, we aren't competitive [a key phrase for MBAs]. So what are the global best practices [another key phrase]? Well, France and Japan have nationalized state-monpoly school systems, operated by civil servants under extremely rigid rules. Long silence ensues.
PS to lizardbreath -- "Agency costs" is a term of art from economics that's migrated into corporate law (via the Law & Economics crowd, I suspect).
I didn't know that girls were able to teach calculus. You learn something new every day.
Emmy Noether is going to rise from the grave and slay your ass.
LB doesn't seem to be around, so I guess I'll tell Steve that it was the Peace Corps that brought her to Samoa.
44: And also probably Zombie Sophie Germain, too.
(Wait, wait... Does that mean that Ogged is like Lagrange? It's suddenly clear to me...)
Hi, I missed the rest of this discussion but I wanted to point out another bad thing about teaching to a test if it is the main way stuff is taught - kids get used to thinking that there is *no way* they can answer a question if it is something "they didn't take" and they are unable to connect stuff. E.g. my friend in Egypt who had the very test based system in school, when she was taking the very onerous H.S. graduating test (thanawiya amma - general secondary) had a question in physics that involved algebra principles. The kids had all had both classes. But the idea of using a principle taught in one to answer a question in another involved them bringing in stuff that "had not been covered" in Physics class. She said many students were in tears because they just could not see how the concept in question could have been easily accessed if they had connected "what was covered" in Algebra 3-4 to "what was covered" in Physics.
In other words it makes students extremely passive. They are actively discouraged (like the Stegosaurus example above) from learning stuff that isn't covered. Then they have to come up with it and they freeze. And, they are not taught to be creative or even to use analytical skills.
Of course you can have better tests. But if the test-based environment is the norm, you get kids who can find actual initiative and creativity extremely hard and unfair and they just sort of shut down.
I am facing this with my kids right now who have been in the Egy. system and now have to perform in the US system which is not nearly as rote based and involves group work and independent research and stuff like this. They are finding it hard but with a lot of support from me and patience from their teachers they seem to slowly be getting the idea. (And they perform quite well on tests and quizzes but this is not a majority of their grade in most subjects they are taking - in fact the biology teacher for my sophomore let him take a test in lieu of the bad grades he was getting on labs because she thought he needed to be slowly acclimated to the vast difference in how science is taught here).
A lot of the best science grad students are East and South Asians who spend years mastering rote material and then come to the US and learn to work more imaginatively and independently. Doing science requires a lot of preparation and mastery of detail before you get to the creative part.
John Emerson: The science programs in E. and S. Asia may be actually spending time in labs. That was the big problem in Egypt. The science teacher in my kids' 5th grade actually told me she WANTED to spend more time in the labs. This was a private school that had plenty of equipment. The problem was no time in order to get the kids prepared for the exam - they were supposed to focus on things like definition and spelling of scientific terms. Spelling. I ask you. She said she had to give spelling quizzes. As they got through middle school and my oldest took 3 science courses last year in his first year of secondary, they still spent very little time in labs. And half the time the teacher conducted the experiments while the kids just watched.
I agree with those who said that tests don't necessarily have to be stupid, and the Egyptian government department of education is probably a worst case scenario, but there it is.
Also, i wanted to tell you that my kids miss the mean streets of Cairo and are very bored and annoyed by the concept of waiting politely for the lights to change instead of wandering across oncoming traffic. :)
Also, I agree that the rote based system does give you a knowledge base. Most kids in Egypt probably know more geographical facts than Americans (and my kids found the "dumb american" Canadian videos so funny because they could not conceive of someone who did not know these types of things). I don't think education should focus on one extreme at the expense of the other (all rote vs. all creative) but rather give kids a grounding in basic knowledge as well as a grounding in analysis and creative thinking. Easier said than done, I know.