See also: NYSUT Guide to the New Standards

Are the Regents tests easier? How does scoring work? New York Teacher poses these and other questions to the SED

The state's rollout of new Regents Exams continues to be a hot topic as teachers and students face new expectations along with the new tests. Questions abound on issues from test design to implementation to scoring. In a wide-ranging interview with James Kadamus of the State Education Department, New York Teacher posed questions on the tests and other issues affecting New York State United Teachers members.

Part one of the interview with Kadamus, deputy commissioner for elementary, middle, secondary and continuing education. Part two, also on this page, focuses on local testing and the use of alternative exams.

PART ONE

New York Teacher: Much concern over the new Regents Exams - particularly the Living Environment Exam - involves students being able to answer fewer than half the questions correctly and still pass. Can you explain, in basic terms, how that can happen?

James Kadamus: The number of questions correct is not as important as the combination of difficult items and easier items. In a test like the Living Environment Regents, there must be a sufficient number of more difficult questions to allow higher achieving students to reach scores of distinction. On such a test, achieving a passing score requires the student to answer some difficult questions even though the number of questions answered correctly may be fewer than half. Remember that the decision about what questions need to be answered correctly to demonstrate a student is achieving the standards is determined through a formal standard-setting process. The standard-setting committees are composed of expert teachers from that subject area.

NYT: The Living Environment test was new last year. Many teachers felt it was "easier" than the Biology Exam it replaced. Are you re-evaluating the "cut" points that determined what scores equal proficiency on this new exam?

JK: We will continue to look at the standard-setting system to see where the cut points should be. We've now had three tests - in June, August and January. After those three administrations we'll look back and see if we need to revisit the system of deciding what levels are proficient, advanced and low-pass. It's possible we'll come back and say students need to get more questions right to be considered proficient ... or we may say no, this was about right.

NYT: So perhaps the system just needs some fine-tuning?

JK: We're in a period of refinement. I don't think we're going to see major changes but we'll continue to watch the data. We're trying a lot of new things, and as we get more data we'll refine the tests and the scoring. Clearly, we need to do a better job of getting more information to teachers, and we're in the process of doing that. We recently sent a field memo to all teachers on the state testing system. We'll be working through BOCES and the teacher centers to establish some followup training on the tests. And we're doing training using teleconferences.

NYT: Tell us about the field memo.

JK: It talks about the basics of the state testing - how we're designing it and what the process looks like. We wanted to define terms and answer basic questions. For example, there are questions about the balance between testing content and skills. (The full memo is available at: www.emsc.nysed.gov/ciai/testing/inform/stoamemo901.pdf)

NYT: The Living Environment Exam seems more skills-based than content-based. Are there plans to design all Regents Exams this way?

JK: One of the things we're trying to do is work on a balance between content and analytical skills. What's influenced us is the information explosion. Even five or 10 years ago, if you were a college student working on a research paper, you probably spent 80 percent of your time collecting the data. You only spent 20 percent of your time actually analyzing the data and writing the paper. Now students can get more information by the push of a button on the Internet than by spending weeks in the library.

Now, their time is spent sorting through a huge amount of information. It involves synthesis, analysis, making judgments and applying that information to solving a problem or research question. Given that fact, the Regents Exams need to have a balance. We know content is important because you can't understand science without speaking the language of science. There's still a lot of content knowledge. But the analytical part is also important.

NYT: Is that making the exams easier for the test-takers?

JK: Some people say that. I disagree. They're saying, "Look, the information is right in front of you." But the skills of interpretation and analysis are more sophisticated and demanding than factual recall. We want the Regents Exams to be more relevant in terms of the needs students are going to have in the future. We think some of those analytical skills are just as important as memorization. Doctors have Palm Pilots so they have immediate access to technical information. Our whole society is moving away from being just fact-based toward a more analytical system. We're trying to have that balance in the Regents Exams. I think teachers who have traditionally looked at it as memorizing content and regurgitating the content on the exam are seeing a change here, and they're going to have to adjust to those changes.

NYT: What are other changes?

JK: As we move from a two-tier system to a system where we expect every student to be involved in a laboratory science, many students will not necessarily be going to go on to college as science majors. Memorizing parts of the anatomy doesn't seem to be the right thing to do for those kids. What we're trying to do is give them a basic set of scientific concepts, a language of science they can use, and apply a method for scientific inquiry. Think about the news stories over the past few months - anthrax, smallpox, environmental concerns, biological warfare. All students benefit when they can apply scientific knowledge to these issues. Think about the articles on anthrax: if you don't know what a spore is you're going to have trouble figuring out why anthrax is a problem.

We cover that in Living Environment. That's the kind of scientific knowledge and scientific method that's important for all our students to have.

NYT: Does this come at the expense of high-achieving students?

JK: No. First of all, those kids should be shooting for much higher than passing. No one's going to get into a pre-med program getting 65 on the Living Environment Exam. Our data show that there is plenty of room at the top. We don't have a high proportion of students reaching distinction on the science exams. Second, high-achieving students really ought to be planning to take more advanced courses. There's nothing stopping schools from saying all of our kids are going to four-year colleges and a lot of them want to be science majors. So why not teach an advanced biology course? They can still go back and take the Regents Exam, but why not push them to a higher level and get into Advanced Placement? The commissioner just approved AP Biology and the SAT II in biology as alternatives to Living Environment; the AP and the SAT II are approved alternatives in U.S.history. You can actually get a Regents diploma in New York state and only take approved alternatives to the Regents. (For the complete list of alternatives, go to: www.emsc.nysed.gov/ciai/testing/hsinfogen/appalt.pdf).

NYT: In some districts, students who were failing the course nonetheless passed the Living Environment Regents Exam.

JK: There's always been a very high correlation between class performance and Regents performance. The intent is that Regents Exams are end-of-course or end-of-multi-course. So I'm concerned when I hear that, because it suggests a potential misalignment - that some of the things teachers are teaching may be at a different level than we're testing here. I don't think it's a sign that the Regents Exams are not rigorous. I think we need to find out more from those teachers about why that happened and then begin to analyze it.

We had a summit last August of all the associations of science teachers. We asked them if they thought we were out of line with the coursework and whether they heard of a misalignment happening on a widespread basis. They said they heard a few examples but it wasn't widespread. By and large kids' class grades pretty well correlate with the Regents Exam. But we'll watch this.

NYT: Some of those scoring the new Regents Exams believe the rubrics lack rigor.

JK: I don't agree. I think that in some cases it may be that people don't understand the application of the rubrics and are giving higher scores than they should be; or, in some cases, the rubrics are applied more rigorously than need be. There's always a sorting-out period. If you remember the first year we did the English Language Arts Exam, there was concern about the scoring rubrics and how to apply them. Now that we're in the third year of Regents English, most people are saying it's a high-quality exam.

NYT: How are you addressing the issue of training scorers, especially for the sciences?

JK: The message we're getting back from teachers is that we need to do more training on the philosophy behind the exams and the scoring of the exams. Obviously we have some limits on resources, but we will do more training statewide and regionally. We had a teleconference in November for science teachers on the scoring of science exams. That gives us an opportunity to provide information that people can videotape and re-use.

NYT: Is this basically a period of adjustment?

JK: I think it is - a period of adjustment and refinement. We could have pursued a different strategy. Massachusetts had a two-year period during which students took tests but they didn't count. Only a little over half the kids were passing the tests. Once the tests "counted," 75 to 80 percent of the kids passed. When things count, there is a different motivation. We conduct pilots and field-test, but it's important to move quickly to the point where the tests count, then refine them as we go along.

PART TWO

NYT: Tell us more about the field-testing you referred to.

JK: We've increased the number of schools and students involved in our field-testing. We've been very precise about our sampling, which gives us more information about how these questions work on a wide range of students. It also enables us to do sophisticated bias analysis that we never did before. An important aspect of the field-testing is using common questions - "anchor questions" - in the different test forms that are field-tested. Anchor questions enable the state to develop the sacale scores before the tests are administered. Through the anchor questions, these scales are all equitable from one test administration to the next. This assures a fair score for all students who take the test.

NYT: Is there a way to simplify the scoring process?

JK: We're all struggling with the complexity of testing. When we did the field memo in September we rewrote it a number of times. We had the NYSUT staff look at it and got feedback from teachers and principals and kept trying to make it simpler. Some people still say it's too complicated. But it's much like taking the Living Environment Exam. You can't understand biology without understanding the vocabulary of biology and you can't understand testing without understanding the vocabulary of testing. We need to do a better job educating all teachers - and the public - about the state testing system. Does it take some work to understand this? Yes. Do we continue to try to make this message simpler? Yes. But we don't want to oversimplify what is a complex process. And we want to make sure that people understand there are certain basic rules that we have to follow in creating state testing that may not apply to their local assessments.

NYT: You've expressed concern about local assessments.

JK: We've been talking with a lot of people both in this state and nationwide, and there's been some research done by the National Research Council that suggests that the state of local assessment in schools across the nation is at an all-time low. The rules of large-scale assessment are pretty rigorous and the degrees of freedom that we have to use different methodologies are pretty limited. A lot of the standards have to be assessed by classroom teachers. What we're finding now is that most classroom teachers are using mini-versions of the state tests as their main means of assessment, and I think they have a much wider range of assessment practices they can take advantage of - writing journals, research projects, classroom observation. We're looking at what kind of guidance we can give teachers. The state test really should be a way of verifying that classroom assessments are on the right track and teaching is at the right level and that the teachers are teaching to a state set of standards and whether schools are meeting some overall statewide standards. We would like to provide more guidance for teachers on local assessments that work.

NYT: That begs the question of SED's refusal to allow some alternative schools to substitute portfolio-based assessments for Regents Exams.

JK: It's important to recognize that there are alternatives to the Regents Exams; you can get your test approved and your approach approved. But we only approve alternatives that we feel have the same or greater rigor and are connected to the standards. We have now approved about 18 alternatives.

NYT: Is there a specific problem with portfolios?

JK: They tend to be internal to the school. That makes them very valuable for classroom assessments, but difficult to be used on a statewide basis. You would have to standardize the tasks, the scoring rubrics and the reporting. It would be very difficult for us to do and I don't think most people would support the idea of the state saying: "This year in US history every student will do a research paper on the Civil War." It's interesting to note that International Baccalaureate tests, which we've approved, are a combination of on-demand reading and writing as well as research projects and portfolios.

NYT: Are there any plans to extend component re-testing beyond English and math?

JK: We may in the future, but right now English and math are where we want to concentrate. We found that the number of kids involved in component re-testing was very small - about 9,000 out of about 150,000 to 170,000 kids who take a typical Regents Exam during the year.

NYT: What happens to the student who just can't pass a Regents Exam, even with the low-pass and component retesting?

JK: Our judgment, based on our research, is that virtually all students who don't have a cognitive disability - who are trying, doing their homework and studying - can, after multiple tries and having both the low-pass 55 and component retesting, pass the Regents Exams. The philosophy behind the standards movement was to create a set of standards that were rigorous - a reach for students but not unreachable. Certainly there are some students with disabilities that limit their performance, and we've given them alternatives. The ultimate answer is if you can't pass the Regents Exam you can't graduate from high school. But it's also true of courses. We're finding that more kids don't graduate from high school because they fail courses than fail the Regents. And the level of that course work is set locally. The state does not guide districts about what the local grading system should be or what the local assessment situation should be. It's really up to teachers to decide whether students are passing the courses.

NYT: How will the new requirements of the federal Elementary and Secondary Education Act affect testing in grades 3-8?

JK: There won't be any additional tests in grades 4 and 8, where we're already doing state testing, but there must be state examsin English and mathematics in grades 3 through 8 beginning with the 2005-06 school year. Those off-grade tests will be aligned with the fourth- and eighth-grade tests but they won't replicate them. We want to create tests that will give us good information about students while we keep testing time to a minimum.

NYT: With SED looking at proposals to revise the eighth-grade testing schedule, when can teachers and students expect some relief?

JK: If there is going to be change in the schedule it would go into effect in 2003. We've already set the schedule for 2002. In general, teachers would like to spread out the administration of tests in middle school and have them during testing periods where students would take them seriously. Some people have proposed giving them during Regents Exam periods so that students will see them more as a part of the state testing process. We have very little agreement on the idea of moving some of the tests to earlier grades. In general, teachers are not in favor of that. They feel that these are end-of-middle-school tests and they ought to be kept at the end of middle school or as close as possible.

NYT: You seem to have been heavily promoting teacher involvement in the development of statewide tests.

JK: One thing teachers have to know is all state tests, whether it's elementary, middle or high school tests, are all developed with NYS teachers. For the elementary and middle tests we work with a testing contractor who creates multiple-choice questions and shows them to teachers, but the teachers pick those questions. The teachers select the ones for use based on the standards. All of the essay questions are developed by teachers. In the Regents Exams that process is managed by the state, not by the contractor, and we bring in committees of teachers. As I pointed out we usually have two committees for a test - one that develops the questions and another that evaluates the results and sets the cut points.

NYT: Where and how do you recruit teachers for this?

JK: We try to get volunteers. We use the various associations: NYSUT has subject-area committees and has offered us a number of names. We consult with associations that represent teachers. We try to get a range of teachers in terms of experience. We also try to get teachers who represent the diversity of the state in terms of geography, ethnicity, gender. It's a difficult task, and if there are people who want to volunteer, they can e-mail us at atemscgen@mail.nysed.gov.

NYT: Do you feel component re-testing is at least as rigorous as the Regents Exam?

JK: It's quite difficult because it's a concentration of work in areas that you're weak in. The full Regents actually has some compensatory qualities. You can do poorly on one section but do great on another section and they kind of make up for each other. When you're doing component re-testing you're zooming in on the area you know you're weak in and you've got to do well on it. On the other hand we put it in because it does allow teachers to focus instruction. It gives kids another opportunity and we want to continue to do it, but it's a relatively expensive proposition for the number of kids involved.

NYT: What's the difference between "curving" an exam and what SED calls "predetermined scaled scoring adjustments?

JK: If you used a curve on a Regents Exam, you'd get all of the data from the exams, look at it and say "Oh, it's too high" or "it's too low" and add or subtract points to it after the fact. Before the Regents are ever given, those teachers have come in and set that cut point based on experience and feedback and their own special judgment. That's the difference when you're setting the cut points in advance, based on professional judgment. It's their judgment as to how many questions you have to get right to meet the standards. I would be against having any kind of curving. Most teachers can remember times when we gave the Regents Exam and the state adjusted it five points up or down -- usually lowered the passing score by about five points because too many kids failed and teachers complained.That hasn't happened recently because I think we have a much better system of setting the cut points in advance.

NYT: Is there anything else you'd like to tell our readers?

JK: We have a technical advisory panel in addition to the teachers who are involved in these tests. The panel has some of the top national experts in testing review our tests at all levels and ask us to produce research documents to show, for example, that the tests measure the standards, that they are valid and reliable, that the scoring is reliable, that the scoring results fit certain patterns. Our tests must meet standards that the American Educational Research Association and American Psychological Association have developed. They're constantly challenging and asking for research reports and we're producing them. We've made those reports public.

There's an incredible amount of checks and balances in this system both in terms of the development of the tests, field testing, the process for setting the cut points and the evaluation of the tests' tests by the external panel. We also have an extensive bias review. We bring in groups, religious groups as well as other community organizations to look at every test question and say "We don't like that question because we think it's offensive toward a race or religion or it would be difficult for a student to answer because he or she lacks experience in that area." We like questions where no one would have an advantage because of previous experience.

It takes about two years to develop an exam. That's why sometimes teachers say I gave you feedback on the exam and how come there isn't anyone listening out there. Sometimes it takes awhile because we're running two years ahead. If you look at the number of tests produced, we produce well over a million exams a year. If we were a commercial entity we'd be one of the top five testing companies in the US.

See also: NYSUT Guide to the New Standards

NYSUT.org. Copyright New York State United Teachers. 800 Troy-Schenectady Road, Latham, New York, 12110-2455. 518.213.6000. http://www.nysut.org. For questions about this web site, contact the webmaster at bthomas@nysutmail.org.