Rule 6A-1.09430: Establishing Achievement Level Standards for
Florida Standards Alternate Assessments
Webinar Transcript
October 26, 2017

(Slide 1) My name is Vince Verges. I’m the Assistant Deputy Commissioner for Accountability, Research, and Measurement. We’re here today for Rule 6A-1.09430, which establishes achievement level standards for Florida Standards Alternate Assessments. This webinar will be recorded, and we can begin recording at this time. And we will take questions only via the chat feature. If you are not in a position where you can provide questions via the chat feature, we will provide links at the end of the webinar where you can provide comments on the rule. You can also provide your questions there, and we will answer the questions if they appear there. And again we will also post the presentation itself as well as the recording on our standard setting website. You can review; there’ll be a number of slides that you may be interested in looking at more closely. They’re here for your information and for our records. And if you want to look at them more closely and look at them for a longer period of time, you can find those, again, that link will be provided at the end of this webinar. At this point I'm going to mute the line so that we don't have any interference. And then questions should come via the chat box or via the questions, State Board rule workshop input link that will be provided at the end of the webinar.

(Slide 2) So the purpose of the workshop today is: we are going to provide you with our intent to develop a rule for the State Board of Education that can establish achievement level standards of the Florida Standards Alternate Assessment (FSAA). And we're going to obtain input if not via this webinar then via your comments or both that will be considered by the Commissioner in determining the recommendation achievement level standards that will be presented to the legislature for their review, and then to the State Board of Education for its action. 

(Slide 3)During this presentation, I will give you an overview, standard setting process overview, a brief overview of the Performance Task components that are going to be up for having the cut scores adopted. Those are in Civics and U.S. History. We will provide information about the standard settings cut scores for Performance Task that were already passed earlier this year. We’ll talk about the assessment, the recommendations from the educator panel for Civics and History, impact data, and the Datafolio, we’ll talk about what that component of the alternative assessment is, recommendations from educator panels and the impact data for that component, and then we'll talk about next steps and then request feedback from you again via the chat feature in the webinar or via the website link that you should be able to write comments in writing. 

(Slide 4)

(Slide 5)As I mentioned, we will first go over the overview. So the FSAA, the Florida Standards Alternate Assessment, measures students’ mastery of the access points and their essential understandings for English Language Arts (ELA) and Mathematics. And to the Next Generation Sunshine State Standards for Science and Social Studies. 

(Slide 6) The assessments measure student achievement of these access points of the Next Generation Science Standards and making the access at the link provided on the screen there; again, the link will be live in the PowerPoint presentation, it’ll be on the standards website. For spring 2016, we had our baseline implementation of our Performance Tasks assessments in all of our ELA, Mathematics, and Science assessments. Those are given in all the grades and subjects that the FSA and the NGSSS assessments are given. And then in spring of 2017, this past spring, we gave the Performance Task assessments for Civics and U.S. History, and the Datafolio for all subject areas: ELA, Mathematics, Science, and Social Studies. The transition from the previous generation of standards and the Florida alternate assessment. Again, more information is available on the DOE website under alternate assessments with the link available there and again on the PowerPoint that will be provided of the standard setting website. 

(Slide 7) The FSAA has two components. It’s one assessment system which is divided into two parts. Most of the students who take the alternate assessment take the Performance Task, and very few students are eligible to take the Datafolio. 

(Slide 8) The Performance Task is designed for students whose participation in the general assessments, that’s FSA or the NGSSS assessments, is not appropriate, even with accommodations. And it measures students’ academic performance on the access points in ELA, Math, Science, and Social Studies. So the Datafolio is again part of the overall alternate assessment component that's designed to address the needs of a very small population of students who typically do not have a formal mode of communication and may be working at a pre-academic levels. It’s aligned with pre-determined standards and submitted via an online portfolio system during three collection periods that occur throughout the school year. The scoring outcomes are designed to show progress along the continuum of access to the content via reduced levels of assistance and increased accuracy. This is all lots of words and language, and we provided some information on the following slides that provides a little bit more about the population that takes the Datafolio, exactly what the Datafolio is, how it’s administered, and how it’s scored. I will say on this slide that the Performance Task, we have around 27,000 students who take the Performance Task each year. And in the past year in 2017, 2016–2017 school year, we had about 600 students who took the Datafolio. So this 600 students represents about 2% of the students who took some part of the FSAA assessment, and it represents only .03% of the students who take all of the statewide assessments. So, again, a very small population of students who take the Datafolio. 

(Slide 9) This assessment checklist that you will see later in the presentation as well helps us determine whether students should take the FSAA or should take the FSA or the NGSSS assessments. So we want three yes answers to these questions: Does the student have a significant cognitive disability? And even with the appropriate allowable instructional accommodations, assistive technology or accessible instruction materials, does the students still require modifications? And lastly, does the student require direct instruction in academic areas based on access points rather than the full set of standards to acquire, generalize, and transfer skills across settings? If the answer to all three of those questions is yes, then the FSAA is the appropriate assessment for the student. 

(Slide 10)

(Slide 11) In this next section, we will give a general overview of what the standard setting process is like. Why are standards necessary? And there are different kinds of standards we’ll discuss in this presentation. We want to know what students should know and be able to do. We want to have clear expectations for the students’ parents and teachers. And we want to use these results and standards to help us improve teaching and learning in Florida schools. 

(Slide 12) Couple of different kinds of standards, and one of them is the content standards. So that is the Sunshine State Standards–Access Points to Next Generation Sunshine State Standards–Access Points. That tells us what the students need to know, what kind of skills they need to have. Then there are achievement standards that describe how much the content of knowledge, knowledge a student needs to demonstrate that they know the material. These achievement standards are broken up into different levels, and the student standards are used to help set graduation requirements or you have a minimum score that’s required for graduation, such as the Access Algebra 1 test or grade 10 ELA assessment. And lastly are the accountability standards. Currently the Florida Standard Alternate Assessment scores are not used in school grading or other federal accountability reporting, but the idea is that in future years these scores will factor somehow into accountability standards for schools and also at the federal level. So that’s still to come, but we expect to use in that way in the future.

(Slide 13) The standard setting technically is deriving level of performance on assessments by which decisions or classifications of students can be made. That's from the professional of literature. We can use test scores to use, to groups student into these meaningful achievement levels. And then the standard setting is where we draw the lines that separates students into these various levels. And we have to do this every time we set new standards on new assessments, whether that’s the alternate assessment or the general education assessment or our teacher certification or educational leadership exams. 

(Slide 14) We set standards based on what students should know and be able to do, not about what students currently know and are able to do. When we set standards, you want to make sure that we set standards that are applicable to all students across the state. 

(Slide 15) Our list is growing longer and longer every year. We’ve done the standard setting many, many times. I won't go through each of these individually. You can look at them more closely if you want to review the presentation later. But we've done this since 1998 and a number of times for both our general education assessments as well as our alternate assessments. And most recently, earlier this year as I mentioned previously, the alternate assessment Performance Task standard setting for ELA, Mathematics, Science, and the EOCs (End-of-Course assessments) in Algebra 1, Geometry, and Biology were conducted and approved earlier this year. 

(Slide 16) Every single time we do standard setting, it’s always a multi-stage process that follows this pattern. We start off with achievement level descriptions, that is, what do we want students to know and be able to do in each achievement level. The achievement level descriptions are given to the educator panel along with recent test results of students who've actually taken the assessment. And these educators make recommendations on where they think the cut score should be, where is the line that separates one set of students from another in each achievement level. That brings us to today's process for this particular part of standard setting: a public input workshop where we provide this information, let you see what the educators have recommended, and get your specific or general feedback on what you think about what the educators have recommended. After this meeting today, and after we collect the input, we're going to provide all this information to the Commissioner for her to make her recommendations on where she thinks that the cut score should be, and she can provide that to the legislators for their 90-day review. And after that 90 days, that material will all be presented to the State Board of Education. The State Board of Education has access to these achievement level descriptions. They have access to the educator panel recommendations, input from public workshops, the Commission's recommendations, and any comments that the legislators may have had on these on the cut scores. 

(Slide 17) Some dates on where we’ve done things along the way here. The achievement level descriptions for this set of tests that we're setting standards on was held last December in Orlando. Our educator panel was held in Tampa, in July. We had a public workshop for the last set of standards on September 6. And then the legislative period is going to be from through October and January of 2018. So we anticipate that this State Board of Education presentation will be January 17th. So this particular public workshop is today, of course, and then that’s for a previous set of standards and so it’s off; so we will change that slide before it's posted. 

(Slide 18) FSAA―Performance Task Overview.

(Slide 19) A little bit about the assessment. Again we set standards on all of those tests you see with the X's that are marked in blue, those standards have already been set and approved by the State Board of Education. And here we're finalizing the last set of Performance Task assessments for Access Civics and Access U.S. History. 

(Slide 20) As I mentioned, we have about 25,000, which is actually closer to 26/27,000 now, participating in the Alternate Assessment―Performance Task, and these are students that I mentioned previously with significant cognitive disabilities. IEP (Individual Educational Plan) teams are responsible for determining whether these students should be assessed on the administration of the general statewide test or the FSAA. And then after it’s determined they are going to take FSAA, it’s determined whether they should take the Performance Task or the Datafolio. And the IEP teams use the assessment checklist to make this determination. And we'll have a demonstration of that in just a moment; you’ll see that. 

(Slide 21) The performance or task-based assessment, is what the Performance Task is, is given to each student individually via paper by the student's teacher, a certified teacher, or someone else who has worked extensively with the student and is trained in administering the assessment. Students will select an answer to a question from three options that are represented by pictures that are paired with text, numbers, or symbols. Students use whatever primary mode of communication they use to make their choices. And the FSAA―Performance Task is an untimed test. 

(Slide 22) When we create these tests, we use what's called Elements of Universal Design to make sure that students have equal access to the items for everyone. Again, this is whether the students have a disability or not, or different disabilities. And students who need additional accommodations can access Braille or tactile materials. They can have one-sided booklets; they have objects replaced with other objects. They can also have American Sign Language used. Whatever they have normally during their course of instruction, they can have on the assessment.

(Slide 23) As I mentioned, the responses are all, the students use only a paper version of the assessment. In past years teachers had to bubble in student responses, and instead of bubbling in response and mailing them away, teachers now enter responses online after the administration’s been completed. So we don't have to worry about having a student at each computer or the teacher does not necessarily need to enter the information during the tests administration; this can be done after the test is completed. 

(Slide 24) Now we’ll get into the specifics of this Social Studies standard setting for the Civics and U.S. History tests.

(Slide 25) As I mentioned, the standards for all the other Performance Tasks were adopted by the State Board back on May 17th in 2017, and then Social Studies assessments in Civics and History were administered for the first time this past spring. 

(Slide 26) As I mentioned, we had an achievement level description panel. This was for this assessment as well. That was also in December of 2016. It was a two day workshop with 24 panelists that were Florida educators who know this population. And they wrote these achievement descriptions of what students should know in each achievement level and what they should be able to do. And the ALDs are a link between the content and the achievement standards. So the full set of standards, or the full set of access points, and the final cut scores are brought together here by the ALDs. 

(Slide 27) Our next step was our standard setting educator panel, as a mentioned was in July over this past summer. That was a two day workshop, had three rounds of standard setting with 15 panelists, which we’ll describe who those panelists were. If not by name, you’ll know where they were from. We divided into two rooms of seven to eight panelists per room setting standards. We had one room for Civics and one room for History. And they recommended cut scores that are based primarily on content, but we did show them impact data after the first two rounds of judgments. And so we said “Here’s, if you set the cut scores here, here’s how many students will be in each achievement level.” So they had that as a point of reference, but primarily they were basing the cut scores based on the content of the assessment. 

(Slide 28) How we did this was what’s called a Body of Work Method. So panelists looked at actual student work, and they made a judgment regarding the achievement level to which the student work most closely corresponded. Each panelist received student work samples, or profiles, that are based on actual Performance Task, actual students score profiles, that represent typical patterns for the items scores at varying ability levels. Again we order them from students who have the greatest need to the profiles that have, that show the best performance. But we didn't make any determinations about where the cut scores should be; we left that to the panelists. And the panelist classify each profile into the achievement level to which they feel it belongs with relation to the standards and in relation to the achievement level descriptions. 

(Slide 29) This is preferred over some other standard setting methodology that is used because it allows panelists to look at samples of actual student work. It’s more than just multiple choice items. You have some more complexity in this assessment. It is especially useful for complex assessments such as the Performance Task. We've used this process before to successfully set standards on other assessments, across other states, and then also the FAA, the old Florida Alternate Assessment, the standards were set using the Body of Work Method. And then we've always seen that it results in defendable achievement level standards, whether it's in Florida or across other states. 

(Slide 30) As I mentioned, the primary tool were these profiles. We had 45 profiles per each course: 45 in Civics; 45 in History. And each profile represents a typical pattern of item scores at that level. Again I mentioned, they were ordered in ability from lowest to highest ability, and every item that the panelist had looked were operational, which mean they contributed to the student’s scores. 

(Slide 31) Here’s an example of the first 10 profiles. So you can see, there are three rounds of judgments. You see the columns for rounds one, two, and three. And then a student does 16 different tasks. And you can see here in Profile 1, you have a student that scored no points, on the seventh task that the student scored a point, and then the 13th task the student had a point. And as you see all the way down to the tenth, you see students on that profile showed a little bit more ability. Still two zeros and only one three, which you see the 10th profile you see a little bit more ability than the first profile. And again there are 45 of those, and the panelists made individual judgments about what level they thought that profile belonged in. 

(Slide 32) I mentioned the three rounds. There was a practice round where panelists were given a rating form that had three profiles to practice categorizing the four achievement levels, which I’ll describe those levels in just a moment. Round one, the panelists began individually looking at classifying them into each achievement level. Round two, there was a group discussion of the first round judgment and then looking at each profile of where folks thought that each profile rated. They went back to independently revise their judgments as they felt appropriate. Round three was similar; they looked at second round judgments again, had some more discussion, and made a third round judgment. The discussions were in a group, of course. But when they came to rate those profiles, they did those as an individual. Round three was also where the impact data showing how many students will be in the achievement level based on those recommended cut scores. 

(Slide 33) I mentioned the levels and what those look like. Very broadly, whether, no matter what subject areas applies, to ELA, Mathematics, Science, and Social Studies, these achievement level policy definitions describe students in each level regardless of subject. I won't read these to you, but you see the words that separate students: students demonstrating a limited level of success in Level 2, satisfactory at Level 3, and above satisfactory at Level 4, success with the Florida Standards–Access Points. 

(Slide 34) We talked about cut scores or separating students into levels. There's some place along that continuum where a student just barely makes it into the next level. So we're looking at what achievement level these students should go into; we're interested in those students who just make the cut. So we’re not typical of a student meeting an achievement level, though just barely they do make the standard. And I like to use the example of Level 4. So Level 4 is the highest achievement level, and we wouldn’t expect that a student who’s at a Level 4 necessarily needs to get every single question correct all the time. But there’s some level, there’s some point at which we think student performances may not be perfect, but it’s really, really good and it fits that definition of an above average achievement or above satisfactory level performance. And that's where the panelists would make their recommendation on what a Level 4 or where the Level 4 cut is.

(Slide 35) Now getting into what the panelists actually recommended.

(Slide 36) So the panels, they were given an orientation or review of the assessment. They looked at the standard setting process using many of the same slides that I showed here. They reviewed the test in their respective rooms. They looked at the achievement level descriptions. They reviewed their profile training sheets, and then they completed their training round of profile classifications, which you’ll recall, that’s when I talked about they had three profiles to classify into one of the four achievement levels. And then they went over those three round and then recommended four achievement levels standard over those three rounds. 

(Slide 37) We’ll talk a little bit about what the characteristics of the group; again we’re required and it’s important that we share this information so that the public and educators can see who was on this committee. But because these slides have a lot of detail, we’re not going to spend a lot of time on them. I’ll tell you what’s on the slide, and then we’ll move on to the next slide. And, again, you can access this presentation via the webinar or via the presentation itself. The slides will be PDF’d for your review, and you can look at them more carefully. And if you have questions about the make-up of the panel, you can provide those either here on in the chat feature or in the feedback form, which we’ll show you the access to at the end of the presentation. 

(Slide 38) You’ll see there’s 15 total panelists: eight for Civics; seven for History. 

(Slide 39) We have our breakdown of male and female. And generally you see more female because, although the, we can’t be representative of the Florida population as a whole, we’re really constricted by being the representative of the education population, which generally has more females than males in it. 

(Slide 40) We have ethnicity. 

(Slide 41) On the district side of panelists. Very large would be your Miami-Dade, Palm Beach, Broward, and so forth. Large would be your Brevards, your Osceola. And then medium, medium-small, and small districts. You see we tried to have representation throughout all the district sizes. 

(Slide 42) Then we tried to get educators from across the state as well. So 15 panelists. So it’s difficult to get all the diversity that you like, but we do our best to make sure we’re representative all the demographics. 

So, I mentioned impact data before, and what does that mean? We showed the percentage of students that would reach each achievement level at each grade and subject. That’s what impact data is. And, again, that was introduced in round three after the panelists had made judgments in the first two rounds based solely on content considerations and where their colleagues had marked students. Again impact data is used in an important context for the panel's recommendations, but it wasn't their final determination. That's not the primary thing that we wanted them to look at. We wanted them to look at content. So the decisions were ultimately content-driven. 

(Slide 43) We'll show you the education, the educator panel’s recommended achievement levels and we’ll have the impact data to help you inform your own judgment. I’ll draw some things to your attention that are obvious, and then I'll offer some ideas as far as what specific feedback or what general feedback you might be able to provide that is welcome and appropriate. Again educator panelists is based on content. You're able to look at the impact data and make your comments based on the content or on the impact. 

(Slide 44) So the Civics and U.S. History are reported on a scale going anywhere from 725 to 875. 800 would be the median score, middle score. Those diamonds represent where the cuts were. So, 783 would be the cut where panelists for Civics thought would be the difference between Level 1 and Level 2. So the lowest score in Level 2 would be 783. So students scoring 782 would have a Level 1, and students scoring 783 would have a Level 2. And similarly, an 804 is the lowest score at Level 3. That's the lowest score that this panel deemed was satisfactory. That's the definition of student achievement level description. And the 824 would be the lowest score Level 4. 

(Slide 45) Moving on to U.S. History, and I’ll toggle back and forth here, you see that 778 is the lowest score in Level 2 and how that compares. So that’s a little bit lower than what the Civics panel recommended. And they did not see, one panel did not see what the other panel recommended. So they did not have these comparisons. So 783 versus 778. Lowest scores in Achievement Level 3 was a 792 for U.S. History. And the 804 for Civics. There’s a difference there. And in U.S. History you had an 812 as the lowest score versus an 824 in Civics. Again, you’ll be able to access these slides and look at them more carefully and closely as time goes on. 

(Slide 46) Moreover, while the period is open for comments which goes all the way to January. So here’s what this looks like graphically and what we, the impact data that we talked about. So, you can see where the differences are, the differences in the cut scores, and what that did for the changes and the differences in achievement levels. So we can see Civics with a higher bar to reach, so to speak. We have many more students in Level 1 as compared to History. You have of a larger number of students in Level 2 than are in U.S. History Level 2 students. And the gold shading we see about an equal number of students that are in Level 3, but the threshold is much higher for Civics. It's harder to get a higher level there for Civics. And then History it seems to be easier. The panelists set a cut score so that you have lots of students in Level 3 and Level 4. As you see here, more than half of the students are on Level 3 or 4 in History for a little over 40% of students in Level 3 and 4 in Civics.

So your recommendation might be you’d like the distribution to look, make them all look like U.S. History. You might think they should look like Civics. You might recommend that they are somewhere in between. So we invite you to look at these either here or when you go back and review the slides later and make your recommendations for what percent you’d like to see each at level. Or that could be, you can make specific recommendations there or a general recommendation such as making them more like History or more like Civics. Again, it may seem like that’s not a helpful comment, but those things are considered by the Commissioner. And although we don’t have the ability to show you the tests, this is the information we have to go on, then you’re welcome to make recommendations from that standpoint. 

(Slide 47) If we take those numbers I was referencing here in this bar chart, when you go to the table, that’s just these numbers put in a table form for you to refer to. So, as you’re making your comments and looking at it, you can compare and look at these two slides, and you see both a table form and a graphical representation of how many students are in each achievement level.

(Slide 48) We also broke it down by gender. So you see the kids for Civics, for all students you can see the percentages and then how that broke out as far as males and females. And then we combine the table there as well; instead of having the table and bar graph on different charts, we just put it on the same slide here. So that’s for Civics. 

(Slide 49) The same representation for History.

(Slide 50) And then you have ethnicity. So these are a little bit harder to see here, but you can see the impact by ethnicity, which is something that panelists generally like to see and the public likes to see; they want to see what the impact is. You can make recommendations based on all of those, any factor that you choose, that you think is important. So there’s Civics broken down by ethnicity. 

(Slide 51) Here’s that same information again because it would be difficult to put the table and the graph in the same slide. We have that broken out in tables.

(Slide 52) And then the ethnicity broken down for U.S. History in a bar graph (Slide 53) and in a table.

(Slide 54) And then we have another way, we just add some of the categories together. And one of the things we like to look at is, what percentage of students are satisfactory, which is Level 3 and above. So you can see the Level 3 and above column, we have 45% of the students 3 and above in Civics, and 60% of students Level 3 and above in U.S. History. So you can make your recommendations based on those numbers as well.

(Slide 55) We compared, for your information, the FSAA―Performance Task compared to other assessments.

(Slide 56) When we first set standards on Civics, there were 61% of the students were Level 3 and above. And the educator panel for the FSAA―Performance Task is recommending cut scores so that we have only 45%. And again, they didn’t have access to this information because we wanted their basis, their recommendation to be based on content. And in U.S History, when we first set standards on the NGSSS U.S. History standards, we had 57% Level 3 and above. And the educator panel for the Performance Task have recommended the cut scores such that we have 60% of the students at Level 3, Level 3 and 4.

(Slide 57) Moving along to Datafolio. We’ll spend some time on this. For those of you who aren’t familiar with Datafolio, it’s a very complex process. It has taken place, been implemented, over a number of years. The recommendation first came to the department in 2013 from the field because educators didn’t feel like the Performance Task, the Florida Alternate Assessment, was measuring students, this very small population of students with the most significant cognitive disabilities. So the Datafolio was developed over the course of a few years and implemented. And it is quite a complex process but was felt that it was a better measurement of what this population knows and can do, more appropriate for them, and provided more information to educators and parents about student performance for these students.

(Slide 58) This portfolio, the Datafolio, is a collection of student evidence. It’s administered during classroom instruction by the student’s teacher. And it’s a focus on the student’s access to the standards with an emphasis also on the student’s primary communication mode. And the goal of Datafolio is to measure a student’s increase in accuracy, answering questions about the standards, and their independence in how they answer those questions over time. Again, we’ll describe more details about this over the next few slides.

(Slide 59) The Datafolio. We’d originally looked at trying to measure more standards, but as time went on, as we looked at the design of the Datafolio, we decided it was more appropriate to have only three standards per content area per course. And for each of the standards, there’s two to three activity choices per standard that the teacher can choose from. And within those two to three activity choices, there’s five to eight opportunities per activity choice. So we have lots of different ways that teachers, knowing their students, can decide how to measure a student’s progress with respect to the standards. There’s three types of evidence that are collected: observation, some student work product, or some of digital recording of the student. Just like the Performance Task, the Datfolio is a one-on-one administration within the classroom environment. Evidence is collected through three collection periods over the year. Then it’s uploaded into an online platform called the Assessment View System, or AVS. That is uploaded for central scoring

(Slide 60) Just a reminder, we’re setting standards for Datafolio on all content areas: ELA, Math, Science, Algebra 1, Geometry, Biology, Civics, and History. So all of the areas were administered for the first time in 2017, and that’s what we’re setting standards for on the Datafolio. So those 600 students that I mentioned earlier, that’s for all of the content areas.

(Slide 61) So there’s a nice graphic here that talks about the process, and we’ll walk through the process. First is determining eligibility for the Datafolio. Then determining the baseline level of assistance; we’ll describe what that is. Then how the assessment is administered, and then how the assessment is scored to get us to our standard setting.

(Slide 62) The first component I mentioned here: determining eligibility. So, who participates in this? I’ve said this before, and it’s the students with the most significant cognitive disabilities, about 600 for the past year. The IEP team determines whether students will be assessed through the general statewide assessment or the Performance Task or Datafolio. I mentioned this previously. And the IEP team uses the Assessment Checklist that we mentioned earlier, and the Datafolio Participation Checklist to make the determination.

(Slide 63) We saw these three questions before. If all three questions are answered yes, then the IEP team uses the Datafolio Checklist on the following slide to determine if the student should take the Datafolio.

(Slide 64) The additional questions to ask: Is the students’ primary communication only through cries, facial expressions, eye gaze, or a change in the muscle tone that needs to be interpreted by a listener, observer – the teacher, the parent, or whomever else? And does the student respond or react to sensory input from another person, but then requires actual physical assistance to follow simple directions? The answer to that should be yes to get to the Datafolio. And then does the student exhibit reaction primarily to stimuli? They can only communicate that they’re hungry, or sleepy, or tire, or uncomfortable. And then there might be this other piece to look at, if applicable, is a student’s previous performance on the alternate assessment provided limited information? Or has the student historically received a score of 20 or less on the alternate assessment?
So if you have answers of yes on the first three, and you have answers of yes on four and five, then it’s probably appropriate that the student takes the Datafolio.

(Slide 65) And which level of assistance, we’ll talk about what that is, level of support provided by the teacher to help the student access the curriculum. We use levels of assistance because it reflects the classroom practices and implementations of supports that are provided to the student by the teacher or someone in the classroom to help the student respond. Levels of assistance really acknowledges the variety of all the teacher supports required for the variety of students with the most significant disabilities to access the curriculum. And it gives some additional mechanism for demonstrating growth for this population.

(Slide 66) And here’s the levels of assistance that we’ve been talking about, and it’s a continuum all the way from non-engagement where the student refuses, for whatever reason, to engage in the activity to physical, gestural, verbal assistance, modeling assistance, to the point where the student is independent. Generally students go sequentially from one to the next. They generally don’t jump, not that it couldn’t happen. And there may be a student who may actually go backwards. But generally, these are the levels of assistance that we talk about in setting the Datafolio levels of assistance and the level from which we’re hoping the student will grow.

(Slide 67) Now we’re in the administering the assessment. We have the Datafolio given, as I mentioned, through three collection periods throughout the year. Collection Period 1 is used to determine the student’s baseline level of assistance. As you see there in the graphic on the right, we have three standards, we mentioned three content standards; we’ll give you some examples. And then we measure those same three standards over each of the three collection periods. So we use it not only for that first collection period for levels of assistance; we’re going to measure those same standards in periods 2 and 3 to determine whether the student has demonstrated growth through increased accuracy or moving to his or her LOA goal.

(Slide 68) So here’s an example of some reporting categories or some Access Point standards. This is an example from Grade 5 ELA. Where, say, can the student summarize a portion of text, such as a paragraph or chapter? Then there’s three activity choices, so the teacher can say “I want the student to identify what happens at the beginning of a story.” Or we can have the student identify what happens at the end of a story, or the sequence of what happens first, next, and last. So there’s an example. You can see the other Access Point standards and other activity choices. The method of assessing the standard is determined based on the activity choices on the blueprint for that standard.

(Slide 69) So this is a very busy slide, but here’s an example of what we’re talking about. You’ve got the whole reporting category, Language and Editing, in the domain of language, the genre of literary or informational passages. And then you have the standard code. And, within that standard, you have Essential Understandings, or Access Points and Essential Understandings. And then they get down to the activity choices: choice one, two, and three. And the examples of what you’re looking for as far as demonstration. Here we see the emphasis on that red box in the purple for the purpose of our Datafolio process. 

(Slide 70) Zooming in on that you can see our activity choices one, for example, is capitalize the first word in a sentence. The student is presented with and read a sentence with three response options. Which word needs a capital letter in the sentence? And then what the student’s response is.

(Slide 71) For collecting evidence, after administering the assessment, the teacher, test administrator, the teacher submits an Evidence Collection Form, which we’ll show you in just a moment, for each of the three standards assessed in each content area. The Evidence Collection Form tells the standard being assessed, the activity choice that was used to assess the standard. and which collection period it is. And the test administrators then note the level of assistance that the student required to respond to the activity and the percentage of accuracy with which the student responded. 

(Slide 72) This is the Evidence Collection Form. It is very difficult to see in this slide. It is something that, if you would like to see an example of, that certainly can be provided. Again, this is just a sample on the slide there. You can see choice number, here in the presentation here is choice two, Collection Period 1, and it was an observational evidence. The student have five total opportunities. The student had an 80% accuracy and the level of assistance was “G” for Gestural. 

(Slide 73) And now the Running Record here. You can see the example; this is the example that everyone gets, “What would you wear if it was raining outside?” Then you can see the three options that are given to the student. The expected response is, you’re looking for an eye gaze on the picture of the raincoat. If the student responds to a picture of an apple, that’s marked incorrect. The level of assistance was Verbal. So there was an example for the administration.

You see here, you’ve got the five opportunities; the teacher asks theses five questions. For each one: what were the response options, what were we hoping for the student to see, what actually did the student respond, whether it was correct or incorrect, and what level of assistance that we had there. 

(Slide 74) And so the scoring, again it’s centrally scored, the assessments are scored after Collection Period 3. The student receives a score on each of the three standards assessed for each content area, and the scores are based on the student’s progress towards the level of assistance goal and the accuracy with which he or she responds to the activity choice. The score assigned to each standard is based on the Datafolio Progress Rubric. 

(Slide 75) You say, “What is that?” Here we have the Progress Rubric. I mentioned those five levels, those five progress scores. You can see each one, I’ll go through; I’ll mention the first one here. A Level 1 would be the student did not meet the level of assistance goal, and there was no progress from Collection Period 1 to Collection Period 3. Or the level of assistance goal is the same as the baseline, and there’s no progress from Collection Period 1 to Collection Period 3. And you can go on through the continuum to level five where the student exceeded the level of assistance goal with an accuracy of 70% or higher, or the student met the level of assistance goal by Collection Period 2 with accuracy and exceeded the level of assistance goal with accuracy by Collection Period 3. It’s very busy, and you can look at this if you are not familiar with this more closely later. We can see how the student measured across each of the standards and what progress score they earned.

(Slide 76) For the scoring worksheet, take those progress rubric scores and then the scorer can enter the information on the scoring worksheet to arrive at the scoring – we’ll get to the standard setting – to arrive at the scoring. 

(Slide 77) 

(Slide 78) We’ll talk about standard setting. We’ll talk about what it took to get to each of those levels based on those progress scores and the score sheet for each standard. I’m going to mention, students each receive a progress score in each of the three standards, and the standard setting will determine how to align the possible combinations of progress scores to each of the achievement levels.

(Slide 79) That workshop was held, the Datafolio ALD panel workshop was held in April of 2017 for a half-day using five panelists, and they described the achievement levels for the content standards to specify what a student should know and be able to do in each of the achievement levels. And again, just like for the Performance Task, the ALDs are a link between the content and the achievement standards.

(Slide 80) Whereas the Datafolio, the Performance Task component had five, I mean four achievement levels, the Datafolio standard setting, the levels are three. A Level 1 is the student does not demonstrate adequate levels of progression toward independently accessing the standards, the Access Points. Level 2, the student has some limited level of success in their progress. And Level 3 is satisfactory level of progress. And so remember, the Datafolio acts as part of the overall FSAA component. We realize the students who are taking Datafolio would otherwise be scoring a Level 1 on the Performance Task. And because those students would consistently be scoring a Level 1, that doesn’t provide us with any information. So if know they would score Level 1 if they took the Performance Task component, if we provide the Datafolio, they may be able to, or they hopefully would be able to provide some level of progress. And we would hope to see students, some may still be in Level 1 on the Datafolio, some we would hope to be in Level 2 and some more in Level 3, and this provides some more information to parents and teachers on student progress. 

(Slide 81) Panelists would determine, were tasked with determining what level of progress towards the skills, knowledge, and abilities to demonstrate a student’s progress to be classified into one of those three achievement levels. As mentioned, Achievement Level 3 is a satisfactory performance on each assessment. But again, even a Level 2 shows that the students made some progress.
 
(Slide 82) The Educator Panel was held in July during the same time as the Civics and History Performance Task standard setting. They also had three phases of standard setting. We had 16 panelists, and they were in their own room apart from the Civics and History folks. We had four panelists per content area. So they each recommended achievement levels based on the score combinations and scoring rubric with a validation by the content area that includes a review of the impact data.

(Slide 83) We actually had a combination of the Body of Work and what’s called the Reasoned Judgment Method. So, the Reasoned Judgment Method is where panelists review and discuss the ALDs in terms of what is expected of the students. And then a full group, facilitated discussion, the panelists talk through the realistic expectations based on the achievement level descriptions, and they listed the progress needed for a student to represent a specific ALD. Panelists examined the scores and they divided the full range of possible score combinations into the desired categories.

(Slide 84) Why do we do this? Because this assessment is just for a subset of students, and they’re at a pre-academic level, access level. We’re looking at students, basing scores partly on meeting progress targets as opposed to standardized levels of performance. So we want to make those judgments prior to looking at student work. So we’re not judging them against a whole group; we’re judging their individual progress. The score combinations, you want to look at those to determine if there are value and weights associated with certain score patterns. And then this Reasoned Judgment allows panelists to parse out the above and that it’s defensible and credible results. So Reasoned Judgment Method was the first part of the standard setting.

(Slide 85) And then we look at Body of Work. They look at student work across the three collection periods for a variety of different student abilities to make a judgment about what achievement level each group fell into. So these Body of Work sets are student samples, much like the Performance Task, that represent the different score combinations that students might receive. And the panelists can validate or change their categorizations made during the Reasoned Judgment phase of the standard setting. 

(Slide 86) Then we combine this with the Reasoned Judgment because, again, panelists saw actual student work. Again, it’s useful for complex assessments such as the Datafolio, and we used it successfully in the past for setting standards. And again, it results in defensible achievement level standards.

(Slide 87) Just like we showed you examples of what the profiles might look like for the Performance Task, here we have an example of what we might see for the profiles for students in the Datafolio. For Standard Entry 1, Entry 2, and Entry 3, we see that Profile 1, the student showed no progress. They had a progress score of 0 for all of those. All the way through the one you see, Profile 12, we had entry of, a student scored a 5 on the progress level on Entry 1, but then scored 0 on Standard Entry 2 and Standard Entry 3. So although unfortunate that the student had zeroes on Standard Entry 2 and 3, the student did get a 5 on Standard Entry 1.

(Slide 88) Similar to Performance Task, we had three rounds. We had a practice round and then rounds one, two, and three. In phase A, round one, two, and three. Looking at the first score combination, panelists independently classified the profile into an achievement level. They were able to discuss first round judgments in round 2, looking at impact data in round 2 and again at round 3. Phase B was when you look at Body of Work sets. So Phase A you’re just looking at combinations. Phase B is when you looked at Body of Work as well. You looked at the actual student, the Datafolio submissions. And then finally, the last phase, panelists reviewed achievement classifications for all the content areas together and shared the rationale with the large group. One of the reasons we looked at all the content areas together again is because this is a fairly small group. We had 600 students across all content areas. So when you spread it across all the content areas and all the different ranges, you want to look at the whole body together to get a clear picture of what this unique population can do across the content areas and the grades.

(Slide 89) So now we can take a look at what the educator panels recommended.

(Slide 90) So, like we did with the Performance Task, we had a general orientation for the student population, review the assessment, review the standard setting process, using many of the same slides we’ve seen here. Panelists reviewed the ALDs, looked at the profile/rater sheet, completed their training round of classifications. Then they reviewed the content area blueprints, standards, and activity choices. Then they recommended the three achievements in their three phases.

(Slide 91) Again, we’ll have demographics. You’ve already seen the nature of the makeup of the committee for the Performance Task, so we’ll go quickly through these slides. 

(Slide 92) We had 16 panelists overall. 

(Slide 93) The breakdown of male and female, (Slide 94) breakdown by ethnicity, (Slide 95) and district size.

(Slide 96) Impact Data. So we wanted to show the percent of students that would reach each achievement level in each grade and subject. And again, as I mentioned, we introduced that during rounds 2 and 3 after panelists made their judgement based on score combinations and after panelists validated judgment in their content groups. The impact data was used as context but, again, it didn’t ultimately determine their recommendations. Recommendations were driven by content and the amount of progress that students made in that content.

(Slide 97) We’ll show you the impact data and the achievement level recommendations for all the grades and subjects for you to inform your own judgment. And again, you can look at impact data and have that factor more heavily than it did for the panelists because we asked them to look at progress towards academic content.

(Slide 98) For Level 1, we can show a chart as we did previously, but the narrative better explains what it takes to get a Level 1. So, Level 1 was for any student that had a combination of scores on the three standards that includes a 1 in at least one standard but does not include a 2 or higher in any standard. So, you can imagine, you’ve seen one 1 and zeroes everywhere else. So that student would be a Level 1. And you don’t see a 2 anywhere. Or any number of ones. You don’t see a 2 anywhere.

A Level 2 is for a student who you see some combination of scores for each of the three standards where you get a 2 in at least one standard. So you may have zeroes for all the other entries, but you have a 2 in one place. That would earn the student a Level 2.

Then the Level 3 for a combination of scores for the three standards that has a 3 or higher in at least two standards. Okay, so you can have any number of combinations of threes and twos and so forth, but you have a 3 or higher in at least two of the standards. 

So, there’s an example at the bottom: A student takes the FSAA Grade 3 Assessment and received a 2, 2, and a 0. Based on the achievement level standards we just described, that would be a Level 2 student.

(Slide 99) Of the students who had scorable portfolios, or Datafolios, this is the percentage of students that would be in each of the achievement levels for ELA, Mathematics, Science, and Social Studies. 

(Slide 100) We have that in a table form, (Slide 101) by gender both in a graph and table form, (Slide 102) ethnicity in a graph form (Slide 103) and table. We’re talking about fairly small numbers here, so when you break it out this way the numbers are even smaller, but we can still make our judgments based on the percentages and you make recommendations based on those percentages.

(Slide 104) Mathematics, gender.

(Slide 105) Mathematics, ethnicity in a graph form (Slide 106) and in table form.

(Slide 107) Science and Social Studies together; again, very small numbers. So we grouped those together, Science and Social Studies, by gender, graph and table.

(Slide 108) Science and Social Studies by ethnicity in a graph, (Slide 109) and in a table.

(Slide 110) And percentage in each achievement level. So we have Level 2 or Level 3, so we can think of Level 2 and Above as students who showed some progress. Whereas if these students had not taken, or had taken the Performance Task, they would have gotten Level 1. So we have about 71% of students, or 70 to 71% of the students showing some kind of progress. And at Level 3, so anywhere between 25 to 37%, depending on the content area, of the students showing satisfactory progress given their mode of communication.

(Slide 111) Next Steps.

(Slide 112) The draft rule we look to present to the State Board of Education in January after the Commissioner’s recommendations and legislative review. The State Board is going to review the educator panel recommendations that I’ve just shared with you, any public input that comes as a result of this workshop or any other review, and the Commissioner’s recommendation. And they’re going to adopt new achievement level standards for the Social Studies Performance Task assessments and the Datafolio Assessments.

(Slide 113) We appreciate the time that you’ve spent with us in covering 113 slides. Hopefully some of them went faster than others. We appreciate your interest, and we invite you to visit the Standard Setting page that I mentioned on a number of occasions. You can provide your feedback on the educator panels’ cut score recommendations. You can click the Feedback Form under the Florida Standards Alternate Assessment. This is the link where you’ll also be able to find a recording of this webinar, and you’ll also be able to find the PowerPoint itself in a PDF version for you to review at your own leisure. 

With that we’ll take a look and see if there’s any questions in the chat box. 

We have no questions in the chat box.

Okay, we don’t have any questions. So with that, again, please visit our website. If you have any questions, you can contact our office. You’ll go to the FLDOE website, and you can find assessments. And you’ll see a link there to provide for other questions. Let’s go back to that slide there. This presentation will be, we’ll share this, and we’ll make sure that all of our alternate assessment and our assessment contacts have access to these links, and we’ll share this out widely so that anyone can make comments if they so choose. Thank you, and have a good afternoon.