Continuing on the June 30th post announcing the new proficiency cut score for this year’s Biology I EOI test … The performance level description for OCCT Biology is found here. The performance level description for OMAAP Biology is here. Both of these documents define what it means for be proficient and advanced (or not) on the Biology EOI exams. If compared to previous versions of the performance level descriptions you will no doubt notice that these descriptions differ very little. As such, if the descriptions of proficient remain much the same, why the dramatic increase in the score required to demonstrate proficiency?
Standard setting is a process designed to insure alignment of test performance with the agreed-upon description. Committees made up of educators and non-educators are responsible for determining OCCT cut scores. In a process that takes place over a number of days the process works more-or-less like this: For any one OCCT test, several sub-committees are formed and each sub-committee works as a group to become familiar with the test and the performance level descriptions. They do this by taking the test and then as a group, discussing the test and their answers on each item. The sub-committees are then presented with the OCCT test items ranked by their relative difficulty. The easiest item, the item that was correctly answered by the majority of students, is first and the committee is asked to determine what is being asked of the student in order to correctly answer that item. For the next test item, the one determined to be harder because fewer students answered it correctly, sub-committee members are asked to determine what makes that item harder that the first one. The process continues in like manner until a determination has been made for each item as to what has to be known in order for a student to answer it correctly. The result is a narrative, if you will, of all the science content and processes we ask students to consider when taking an OCCT Assessment. Sub-committees are then asked to look at this narrative and find the point where the description for proficiency is achieved. In effect, assuming that students will only be able to correctly answer those items they know, if they correctly answer the first (easiest) question, how many questions will they need to answer correctly to meet the performance level descriptor for proficient? Once each sub-committee has made this determination, the larger committee re-assembled and a process of consensus building begins. This process includes information about the impact of passing rates on any one cut score. But in the end, through a deliberate, organized process, a cut score determination is made based primarily on a comparison of the performance level descriptors and the actual OCCT test.
Through this process one would expect that without a substantial change in the test items or a substantial alteration of the performance description, the cut score would be very similar year to year. But a process is also underway to not only increase the depth of knowledge of the OCCT tests (now 40% depth of knowledge 3 up from about 15% DOK 3 just a few years ago), but to increase the performance expectations as well. This year it is science, next year will be social studies, followed by ELA and then Math in subsequent years.
Slide from Biology 1 Standard Setting report to the State Board of Education, June, 2013
So now we go from a proficiency cut score of 30 or 32, depending on the form version, to a cut score of 42 in one year. This is truly high stakes testing and the real challenge for districts is to devote resources to science after years of neglect due to the emphasis on ELA and Math brought on by NCLB and the Common Core State Standards and the paucity of funding to Oklahoma schools since the start of the Great Recession. To produce students able to meet the science process understandings on PASS, C3, Oklahoma Academic Standards (OAS) or provide for opportunities to engage in the Practices of Science in the Next Gen Standards requires much more commitment on the part of the State and school districts than most of us have seen over the last few years. Can we bring the proper attention to the issue to get the resources we need? Will community folks like the Oklahoma Business and Education Coalition (an interesting “coalition” in that there are no educators on the Board), help get attention to the desperate needs of Oklahoma science teachers? Will the various Oklahoma STEM advocacy groups (again, all business folks, few, if any, educators) step up? Or will 50% of our students scoring Unsatisfactory or Limited Knowledge just result in another round of finger-pointing and castigation of schools and teachers?
More on the New EOI Cut Score
Continuing on the June 30th post announcing the new proficiency cut score for this year’s Biology I EOI test … The performance level description for OCCT Biology is found here. The performance level description for OMAAP Biology is here. Both of these documents define what it means for be proficient and advanced (or not) on the Biology EOI exams. If compared to previous versions of the performance level descriptions you will no doubt notice that these descriptions differ very little. As such, if the descriptions of proficient remain much the same, why the dramatic increase in the score required to demonstrate proficiency?
Standard setting is a process designed to insure alignment of test performance with the agreed-upon description. Committees made up of educators and non-educators are responsible for determining OCCT cut scores. In a process that takes place over a number of days the process works more-or-less like this: For any one OCCT test, several sub-committees are formed and each sub-committee works as a group to become familiar with the test and the performance level descriptions. They do this by taking the test and then as a group, discussing the test and their answers on each item. The sub-committees are then presented with the OCCT test items ranked by their relative difficulty. The easiest item, the item that was correctly answered by the majority of students, is first and the committee is asked to determine what is being asked of the student in order to correctly answer that item. For the next test item, the one determined to be harder because fewer students answered it correctly, sub-committee members are asked to determine what makes that item harder that the first one. The process continues in like manner until a determination has been made for each item as to what has to be known in order for a student to answer it correctly. The result is a narrative, if you will, of all the science content and processes we ask students to consider when taking an OCCT Assessment. Sub-committees are then asked to look at this narrative and find the point where the description for proficiency is achieved. In effect, assuming that students will only be able to correctly answer those items they know, if they correctly answer the first (easiest) question, how many questions will they need to answer correctly to meet the performance level descriptor for proficient? Once each sub-committee has made this determination, the larger committee re-assembled and a process of consensus building begins. This process includes information about the impact of passing rates on any one cut score. But in the end, through a deliberate, organized process, a cut score determination is made based primarily on a comparison of the performance level descriptors and the actual OCCT test.
Through this process one would expect that without a substantial change in the test items or a substantial alteration of the performance description, the cut score would be very similar year to year. But a process is also underway to not only increase the depth of knowledge of the OCCT tests (now 40% depth of knowledge 3 up from about 15% DOK 3 just a few years ago), but to increase the performance expectations as well. This year it is science, next year will be social studies, followed by ELA and then Math in subsequent years.
Slide from Biology 1 Standard Setting report to the State Board of Education, June, 2013
So now we go from a proficiency cut score of 30 or 32, depending on the form version, to a cut score of 42 in one year. This is truly high stakes testing and the real challenge for districts is to devote resources to science after years of neglect due to the emphasis on ELA and Math brought on by NCLB and the Common Core State Standards and the paucity of funding to Oklahoma schools since the start of the Great Recession. To produce students able to meet the science process understandings on
PASS,C3, Oklahoma Academic Standards (OAS) or provide for opportunities to engage in the Practices of Science in the Next Gen Standards requires much more commitment on the part of the State and school districts than most of us have seen over the last few years. Can we bring the proper attention to the issue to get the resources we need? Will community folks like the Oklahoma Business and Education Coalition (an interesting “coalition” in that there are no educators on the Board), help get attention to the desperate needs of Oklahoma science teachers? Will the various Oklahoma STEM advocacy groups (again, all business folks, few, if any, educators) step up? Or will 50% of our students scoring Unsatisfactory or Limited Knowledge just result in another round of finger-pointing and castigation of schools and teachers?