Under half of students projected to test well

Replies

• Gary Ravani 9 years ago9 years ago

  Navigio: A bit of a historical digression and context re the NAEP and CA here. As I recall 1992 was the first year the NAEP scores were presented broken out by state. I attended a meeting at that time with the NAEP people (Sacramento I think) where they explained that they were hesitant to do this, but under political pressure to do so, because they understood that the test was not designed to compare state to state … Read More

  Navigio:

  A bit of a historical digression and context re the NAEP and CA here.

  As I recall 1992 was the first year the NAEP scores were presented broken out by state. I attended a meeting at that time with the NAEP people (Sacramento I think) where they explained that they were hesitant to do this, but under political pressure to do so, because they understood that the test was not designed to compare state to state and that such comparisons would not be “reliable and valid”, but inevitably the press and public were going to do that anyway. The NAEP person appeared to genuinely grieved about that.

  Within a few days the scores were published and CA “compared” poorly to other states and sure enough the hue and cry went up about “The Crisis in the Schools!!!” Coincidently, or not, that was a strange time in CA politics when the perennially minority party was in full control of state government holding the governorship and the legislature. (As a result of that “brief time” the public “learned” and they have remained a minority party ever since.)

  This whole incident kicked off the “standards and accountability” movement in CA. State tests were mandated quickly followed by standards and “accountability.” Meaning based instruction was out and “back-to-basics” was in because, it must be constuctivism that was the culprit. A law was passed outlawing reading programs that were not phonics based.

  This all sounds like it makes some sense except for one thing. The Legislative Analyst’s Office did a report at that time, I think was called The CA Report Card that did analysis of things educational. The head of the LAO at that time issued a report on the NAEP results. The conclusions were that there were really two sets of scores, statically two “J” curves, that showed that schools with students predominately English speaking and middle-class had NAEP scores that matched high scores from anywhere in the nation. Schools with students who were non-native English speakers and poor had some of the lowest scores in the nation. It was only when the scores were aggregated that CA looked bad. Low scores were not a function of teaching methodology, teaching effectiveness, or curriculum they were a function of CA (even then) having relatively high proportions of ESL and poor kids. The LAO report was ignored.

  I was on a local radio show debating the then “State Secretary of Education” (a Democrat appointed by a Republican Governor which was interesting) who was calling for stiff sanctions for schools that didn’t perform. I brought up the LAO report that said the main issues were ESL and poverty not school performance. The Secretary said no such report existed. The debate/discussion kind of went downhill from there. The next day I faxed the report to the radio station and they had me on again to talk about it. I don’t believe they received a response from the Secretary’s office. I do not believe the LAO did “Report Cards” much after that.

  (Below)

  • Gary Ravani 9 years ago9 years ago

    The API was conceived at about this time. The title of the original task force working on this was "Rewards and Sanctions," a name that ended up being politically unpalatable. So in 1992 we had middle-class students who compared favorably in NAEP scores to other high scoring states, but since that time even those scores have declined. (As far as I know.) This could be attributable to a number of things 1) the test and … Read More

    The API was conceived at about this time. The title of the original task force working on this was “Rewards and Sanctions,” a name that ended up being politically unpalatable.

    So in 1992 we had middle-class students who compared favorably in NAEP scores to other high scoring states, but since that time even those scores have declined. (As far as I know.) This could be attributable to a number of things 1) the test and standards based accountability systems enacted in the early to mid-1990s were pretty much a flop if NAEP scores are important to you; or, 2) the content standards adopted by CA (labeled “world class” by many-Fordham for example) called for teaching content that is not well covered by NAEP. My guess it’s some combination along with the tremendous growth of the non-native English speaking population in the schools, and that the severe and chronic underfunding of the state’s schools was really beginning to have an exponentially increasing negative impact after per pupil K-12 expenditures dropped below the national average in 1985.

    But (!), and there’s always a “but,” there’s this: I have contact with direct representatives of tens of thousands of teachers up and down the state including the two biggest school systems in the state. Most of these folks are classroom teachers themselves. I have asked them if any were aware of their respective districts having students participate in the NAEP. Not one was aware of it. I know that many CA districts were resistant to having students participate in more testing of any kind. I heard “rumors” that the USDE was struggling to get a representative sample of students in CA. I also heard that some districts, in order to qualify for some kind of federal “largesse,” agreed to have some kids take it. I know of no one who knows of students taking the NAEP or any agreements with USDE.

    Now it’s possible for kids to be shuffled off without many teachers being aware of it to take these tests. Possible but not probable. Is NAEP getting a true representative sample of CA students to make valid and reliable reports on state performance????

    Note: Let me mention here, for those who obsess about various international test scores, that disaggregation of scores shows US school with less than 10% “free and reduced lunch” have the highest scores in the world. School with less than 25% FRL are the equal of any other nation in the world in the international tests. The obvious “school reform” to bring all of our schools into the highest plateaus of test scores is to insure that we have no more than 10% of our kids who live in condition that would qualify them for free and reduced lunch.

    • navigio 9 years ago9 years ago

      Thanks Gary, I totally agree. Even though i've used naep data in the past, i generally qualify it as being suspect. I should have qualified it here even more. In this case, my underlying point was that the sbac numbers are for a bunch of states combined. we may score differently than them, and clearly we had different standards than them, so i think its important to recognize that we should not be surprised … Read More

      Thanks Gary, I totally agree. Even though i’ve used naep data in the past, i generally qualify it as being suspect. I should have qualified it here even more. In this case, my underlying point was that the sbac numbers are for a bunch of states combined. we may score differently than them, and clearly we had different standards than them, so i think its important to recognize that we should not be surprised if our eventual ‘results’ end up being different than the sbac predications as a whole. 🙂 perhaps even significantly so. And fwiw, those invalid state comparisons can also be done in a disaggregated manner, which would hopefully limit the invalidity to the representation domain.. 😉 that of course should be enough.. 🙂

Replies

• Doug McRae 9 years ago9 years ago

  Gary -- Well, I guess, there comes a time we respectfully have to disagree given our differing perspectives on the issue. Re "all about instructional purposes," I'll agree instruction is the most important purpose but the "all" qualifier reminds me of the 100 percent target for NCLB . . . . setting 100 percent or all conditions is problematic. Data measuring the results of instruction also deserves some time. My personal guideline is that tests to … Read More

  Gary — Well, I guess, there comes a time we respectfully have to disagree given our differing perspectives on the issue.

  Re “all about instructional purposes,” I’ll agree instruction is the most important purpose but the “all” qualifier reminds me of the 100 percent target for NCLB . . . . setting 100 percent or all conditions is problematic. Data measuring the results of instruction also deserves some time. My personal guideline is that tests to measure the results of instruction (i.e., accountability tests) should take no more than one percent of instructional time, or roughly 9 hours per year. Accountability tests get pushback from the trenches when they take longer, deservedly so.

  I’ll agree instructional tests are extremely important for good instruction. Some classroom testing specialists claim good instruction involves 1/3 time doing assessments, for instance good formative assessments (or formative assessment practices) and good interim tests, and I have no reason to argue with that claim. But, my view is these types of tests should be totally under local district / local school / teacher control, rather than part of a statewide assessment system. They should be closely coordinated with other resources to support good instruction, like appropriate instructional materials and quality professional development. They do not need to be closely coordinated with centrally controlled statewide assessments, and in fact when they are close to statewide assessments then the tend to become agents for teaching-to-the-test over time. That circumstance harms good instruction.

  Finally, holding up matrix testing as a preferred method for accountability assessments is problematic. It would be OK if the only target is state level data, or perhaps statewide and data for the largest districts in the state. But matrix testing will not yield accountability data for medium or small districts, or at the school level. Folks want accountability tests to apply to all districts and schools in the state (with perhaps the exception of very small units with less than 100 students), and with that feature on the list for accountability tests, matrix tests won’t work. CA has gone through this analysis several times the past 20 years, and each time the answer comes back that matrix testing designs will not comply with what is desired from statewide tests . . . .

  • navigio 9 years ago9 years ago

    I expect that as we de-tooth accountability tests, increased emphasis for accountability purposes will be placed on instructional tests. As long as public opinion remains any kind of driver, I expect there is no way around that, other than avoiding de-toothing in the first place (something that also has it's dangers). I'd love to be able to agree with you that calling out misuse is a solution, but in reality people care more about the … Read More

    I expect that as we de-tooth accountability tests, increased emphasis for accountability purposes will be placed on instructional tests. As long as public opinion remains any kind of driver, I expect there is no way around that, other than avoiding de-toothing in the first place (something that also has it’s dangers).
    I’d love to be able to agree with you that calling out misuse is a solution, but in reality people care more about the number than they do about understanding what it really means. This is probably one reason we periodically have these accountability ‘resets’ that appear to be designed in a manner that displays growth of any kind.

    • Doug McRae 9 years ago9 years ago

      Navigio — I won’t disagree with your analysis that the public doesn’t understand . . . . I guess I would say this is where a media outlet like EdSource plays a valuable role, an initial outlet for educating publicans who don’t have access to the insider baseball being played with public sector dollars.

  • Gary Ravani 9 years ago9 years ago

    Doug; If there was a chance you were going to be appointed the "testing czar" I'd be inclined to rethink the whole issue of "accountability testing." I respect your intent and analysis. However, that is not likely to happen. I dislike having to repeat the same things over and over as some of the more fevered minds do, but again I point to the National Research Council's assertions that the accountability driven reform movement in vogue since … Read More

    Doug;

    If there was a chance you were going to be appointed the “testing czar” I’d be inclined to rethink the whole issue of “accountability testing.” I respect your intent and analysis. However, that is not likely to happen.

    I dislike having to repeat the same things over and over as some of the more fevered minds do, but again I point to the National Research Council’s assertions that the accountability driven reform movement in vogue since NCLB, and in vogue in CA even longer, have done almost nothing to improve learning in the US.

    I would assert that’s because the performance of the schools has not been the cause of poor performance by students, it’s been poor performance of the state (both US and CA) in dealing with conditions of poverty that are really the basis for low measured student achievement.

    One consequence of this societal inability (or distraction from) the “poverty man behind the curtain” is that the policy elites, both conservative and liberal, are now faced with either admitting that their proposed “solutions,” testing, accountability, charters, technology, etc., have been abject pedagogic failures or they need to find a scapegoat. We know which way they jumped. Hence, Vergara and other more direct attacks on teachers and teachers’ unions.

    Let me repeat something else here one more time: The US has just about the highest rate of childhood poverty in the industrialized world and CA has one of the worst rates for childhood poverty and child homelessness in those same 50 US states. It is foolish to suggest there are not real consequences for allowing those conditions to exist and exist for decades.

    Let me complete this statement by quoting from a recent letter sent to the Obama/Duncan USDE from the Advancement Project, Mex-American Legal Defense Fund, the NAACP, the Lawyer’s Committee for Civil Rights, the League of United Latin American American Citizens, the National Opportunity to Learn Campaign, the National Urban League, the NAACP Legal Defense and Educational Fund, the National Council on Educating Black Children, the National Indian Education Association, and the Southeast Asian Resource Action Center:

    “The current educational accountability system has become overly focused on narrow measures of success and, in some cases, has discouraged schools from providing a rich curriculum for all students focused on 21st century skills they need to acquire.This particularly impacts under-resourced schools that disproportionally serve low-income and students of color.”

    Different take than Vergara, huh? And from people who know.

    You have suggested that accountability tests can be used effectively. Cambell’s law says that, as power inevitably corrupts, attaching high stakes to “data” inevitably corrupts the data as well as the system that uses the data. If people with your intentions and integrity were in charge, Doug, perhaps the corruption could be avoided. But people like you are not in charge. More’s the pity.

    • don 9 years ago9 years ago

      That’s right. The people in charge are instead Brown, Torlakson and the ultra-liberal legislature – the people who the unions put in charge and whom you support.

    • FloydThursby1941 9 years ago9 years ago

      Gary, if poverty prevents success why do Asians in poverty do well? Why is Lowell, 41% free and reduced lunch, one of the best high schools in the state statistically? And if you do guarantee no one with kids is in poverty no matter what, could it spiral downward by making those on low wages work less hard, fewer hours? Taxes would have to go so high on the rest of us … Read More

      Gary, if poverty prevents success why do Asians in poverty do well? Why is Lowell, 41% free and reduced lunch, one of the best high schools in the state statistically? And if you do guarantee no one with kids is in poverty no matter what, could it spiral downward by making those on low wages work less hard, fewer hours? Taxes would have to go so high on the rest of us it would make many people work less. There is a cost to having zero poverty. You see it in Europe. I’m not saying I wouldn’t trade a lot of things with Europe. but this idea kids in poverty can’t succeed when we are paying billions to pay for school breakfast and school lunch, and no deduction is made to food stamps for the lunches so that is extra calories is beyond me. No one is starving. Kids who say they’re just too poor to do well in school are watching TV, on cell phones, playing games, over 40 hours on average. How do Asians do it? If they do, how is it impossible? Why say we have to solve poverty first? Maybe solving academic laziness would solve poverty, not the other way around.

  • navigio 9 years ago9 years ago

    Doug, I think your last paragraph here is extremely important. This sub-thread has evolved to the question of why we do this in the first place. As you pointed out elsewhere, one reason is so we can measure effectiveness of programs and strategies. But when you say that people want accountability to be applicable down to the school level, is the goal really so that we can evaluate those schools based on their particular strategy … Read More

    Doug, I think your last paragraph here is extremely important. This sub-thread has evolved to the question of why we do this in the first place. As you pointed out elsewhere, one reason is so we can measure effectiveness of programs and strategies. But when you say that people want accountability to be applicable down to the school level, is the goal really so that we can evaluate those schools based on their particular strategy of imparting content? Our star results never came with teaching methodology addenda or a link to the district’s curriculum. In fact, few people even knew what was in our standards. Our results largely measure societal influences, and it seems unless there is a concerted effort on the part of test designers, that it may be impossible to distinguish those influences from what we really hope to measure. Of course there have been attempts at this in that we disaggregate scores by ‘circumstance’, but that seems a statistical abstraction more than than a tangible differentiator at the level we’d like it to apply. There is also the similar schools API measure, and perhaps that is getting closer to where we’d like to be but even that has some real problems, not to mention few people seem to take it seriously.
    If you think about it, the school site council is one place where decisions could be made to change programs based on results, but from experience, school level summative data seems much too variable to rely on for that purpose. In theory, local BoEs would be another, but they don’t generally get involved in the specifics of instructional methodology, except at the broad level, eg things like project based learning and the like, but even those decisions seem less based on ‘results’, except when the goal is to close a school.
    Is there anything we are going to do differently in this round such that such ideals can be realized? Aren’t such distinctions critical for local school and district leaders? And is there some kind of guideline that could be used to inform those leaders so their decisions come from an appropriate context?
    IMHO, it even seems that the new process is being designed in such a way as to be applicable even more to staffing questions than the previous one was. Do you think that’s true? And does that come at the expense of program and instructional methodology evaluation?

    • Doug McRae 9 years ago9 years ago

      Navigio -- A few observations on your comment: Re "STAR results didn't come with teaching methodology" -- STAR results were widely analyzed at the local level using local teaching methodology as the context, and that is the appropriate way to include teaching methodology considerations. Re "STAR results measure largely societal influences: -- Yes if view just in terms of overall status, No in terms of improvement data or identifying success stories or pockets for improvement. Re the API … Read More

      Navigio — A few observations on your comment:

      Re “STAR results didn’t come with teaching methodology” — STAR results were widely analyzed at the local level using local teaching methodology as the context, and that is the appropriate way to include teaching methodology considerations.

      Re “STAR results measure largely societal influences: — Yes if view just in terms of overall status, No in terms of improvement data or identifying success stories or pockets for improvement.

      Re the API similar schools measure — Comparative analysis against similar schools is a good concept, but it had flawed methodology in its application via APIs over the past ten years, a changing 100 supposedly similar schools from year-to-year that frequently wasn’t a credible similar schools group. I wouldn’t throw out the concept, but I’d recommend a revised methodology.

      Re your last sentences about the new process being more applicable to staffing decisions and/or program and instructional evaluations — accountability tests are potentially more applicable to both of those areas than pure instructional tests, mostly due to time of testing (end-of-year) and test security conditions. But we have to be very careful with use for staffing decisions, much moreso at the classroom level than at the entire school level. The fundamental problem is trying to associate any given student score with any given staff member, a problem much more severe at the classroom level than at a whole school level, rather than a problem per se with the tests. I wrote an op-ed on this issue several years ago. A dated quote is right on point for utility of instructional tests for the purposes you cite — in 1953, Henry Dyer said “If you use test exercises as an instrument of teaching, you destroy the usefulness of the test . . . . for measuring the effects of teaching.” That is a powerful statement why tests for instructional purposes do not produce quality data for either staffing decisions or program/instructional evaluations, and why folks want accountability tests for these purposes. Of course, the new process right now in CA’s plan for spring 2015 is neither a well designed instructional testing effort nor a well designed accountability testing effort, so right now all bets are off for good use either for instructional purposes or accountability purposes.

      • navigio 9 years ago9 years ago

        Thanks Doug. I don't know what "STAR results were widely analyzed at the local level using local teaching methodology means. I've seen millions of star results, but have never once seen a statement of how any of those results correlated to a particular teaching methodology in the school or classroom at question. Nor how one methodology compared to another and what impact that should have on a participle set of scores. The closest we get … Read More

        Thanks Doug.

        I don’t know what “STAR results were widely analyzed at the local level using local teaching methodology means. I’ve seen millions of star results, but have never once seen a statement of how any of those results correlated to a particular teaching methodology in the school or classroom at question. Nor how one methodology compared to another and what impact that should have on a participle set of scores. The closest we get to that seems to be labeling a school as charter or non, or quantifying credential rates and similar, but those are not a specific teaching methodology. As far as I know, ETS doesn’t have access to that information either, so I expect scale score generation does not include those differences. (Please correct me if I’m wrong, that clearly would be useful to know!)
        Regarding improvement data vs overall status, point taken. However, that’s only under the assumption we disaggregate appropriately. I have seen many examples of ‘pockets of improvement’ (or decline) that turned out to be unnoticed demographic changes. And given the extent of variability I have seen within individual schools or classrooms, my expectation is that we probably still don’t have our disaggregation properly implemented (surely insufficient to separate out competing characteristics, perhaps missing some altogether). To the extent that is true, even improvement data, particularly on small sets, will be of questionable validity, IMHO of course.
        And thanks for the reference to your earlier post. It seems the first of the items you listed has been half implemented, though maybe the caltides portion would be possible via public data, a la latimes (not a statement of validity, just a recognition of an attempt to do this without caltides).
        The second has been implemented now with SBAC.
        And the lack of the third is probably only a concern for insiders (doesn’t seem to matter for court cases, for example). I have also seen calls for assigning attribution to even district admins, who clearly have zero direct connect time, yet are still seen as partly responsible for the performance of the students. Some of the teacher/staff eval laws even attempt to codify these ‘ratios of responsibility’, some even down to custodial and clerical staff.
        I expect to a large extent school and district staff will simply have to live with a measure of inaccuracy or ‘unfairness’ if or when such systems are implemented, just as states and districts now seem more than willing to criticize the validity of the new measures while taking money in exchange for implementing evaluation systems based on those very scores. 🙂