Slide 1
Good afternoon, and welcome to our webinar during which we will be sharing information about the public review of Civics Instructional Materials and the Civics End-of-Course Assessment Test Item Specifications. My name is Vince Verges, and I am an Assistant Deputy Commissioner in the Division of Accountability, Research, and Measurement at the Florida Department of Education. One of the primary responsibilities of our office is for oversight of all of the state’s K-12 statewide summative assessments, including the Civics End-of-Course Assessment. 
I’m joined today by Michael DiPierro, the Social Studies Specialist for the Bureau of Standards and Instructional Support. Michael will be providing you with information on the Civics Instructional Material Review in a few moments.
This webinar is being recorded and it will be posted to the Civics review webpage for anyone who wishes to view it at a later date. The URL for the review webpage will be included at the end of this presentation. For those viewing the webinar live today, you may input questions into the webinar chat feature at any time, and Mike or I will answer those, if possible, at the end of the webinar. Answers to all questions we receive will be posted along with today’s presentation, the webinar recording, and the transcript of this webinar. Questions may also be submitted via email at CivicsReview@fldoe.org any time during the review window, which opened on August 15th and will remain open through October 15th. The email address will also be included at the end of this presentation.
Slide 2
First, I will be sharing a brief summary of House Bill 807, which is the legislation that requires these reviews to take place. Next, Mike will share information on the instructional materials review, then I will discuss the part of the legislative requirement pertaining to the public review of the Civics End-of-course Assessment Test Item Specifications. Hereafter, I will refer to the Civics End-of-Course Assessment simply as the “Civics EOC,” or “EOC,” and I will refer to the Civics End-of-Course Assessment Test Item Specifications simply as the “Specifications.”
More specifically, I will be sharing information on the history of Civics EOC legislation; background of the assessment; its structure and content; as well as the structure and content of the Specifications. I will also review the specific requirements of House Bill 807 as they relate to the Specifications, and I will provide an overview of the structure of this portion of the public review. 
Slide 3
During the 2019 legislative session, the Legislature passed CS/HB 807, which amends section 1003.4156 of Florida Statute, to require, among other things, that the Commissioner work with specified organizations to review all Civics instructional materials and Specifications by December 31, 2019, and statewide Civics course standards by December 31, 2020.
The statute reads, in part: “After consulting with such entities and individuals, the commissioner shall review the current state-approved civics education course instructional materials and the test Specifications for the statewide, standardized EOC assessment in civics education and shall make recommendations for improvements to the materials and test Specifications by December 31, 2019. By December 31, 2020, the department shall complete a review of the statewide civics education course standards.”
Slide 4
The statute states that consultants may include, but are not limited to: Florida Joint Center for Citizenship; Bill of Rights Institute; Hillsdale College; Gilder Lehrman Institute of American History; iCivics; Constitutional Sources Project; educators; school administrators; postsecondary education representatives; elected officials; business and industry leaders; parents; and the public.
Slide 5
Slide 6
Civics instructional materials was reviewed as part of the social studies state adoption completed in 2016-17.
Review based on Florida course descriptions and alignment to course standards. 
Reviews and recommendations for state adoption of materials completed by state experts, including university faculty, K-12 district content experts, K-12 educators, and stakeholders.
Each material was reviewed by two expert reviewers. If the first two did not agree on recommendation, a third reviewer would act as tie-breaker.
Slide 7
Only materials that rated at least 90% of the subject-specific standards as Good Alignment or Very Good Alignment were recommended for adoption.
Per section 1006.34(2)(a), Florida Statutes, the Commissioner is responsible for selecting and adopting materials.
Five materials were selected for state adoption in M/J Civics education (course #2106010).
Slide 8
Review process will take place on the online review platform EdCredible at floridacivicsreview.org. 
Materials are reviewed in three areas: 
•	Content - Content is comprehensive, accurate, and developmentally appropriate for the intended audience.
•	Presentation - Presentation evaluates how the materials are designed to be used in the classroom.
•	Learning - Learning evaluates the effectiveness of the materials in facilitating the learning process, such as motivational strategies, teaching the “big ideas,” guidance and Universal Design for Learning (UDL) principles.

Slide 9
Materials are also reviewed on alignment to each standard for the course. 
Each set of materials submitted for adoption is evaluated based on a scale of 5-1 against the criteria:
•	5:  Very Good Alignment  
•	4:  Good Alignment   
•	3:  Fair Alignment  
•	2:  Poor Alignment
•	1: Very Poor/Not Aligned 
Slide 10
HB 807 call for review of Civics instructional materials will follow the same 5-point scale, with opportunity to give comment; comments are encouraged
Ratings for standards should be based on the materials’ alignment to standards as written. 
Opportunity to comment on the standard itself will be collected during the standards review as a separate process.
Slide 11
All materials being reviewed are current state-adopted civics education instructional materials, meaning, as a condition of adoption, the department and publishers of the materials are currently under a five year contract. 
Please be sure to suggest improvements for materials going forward in the next adoption in the comments section of your review.
Slide 12
•	Review of state-adopted instructional materials open August 15 and are due by October 15.
•	Commissioner to give recommendations for improvements by December 31, 2019.
•	Review of Civics education standards to begin in 2020, and recommendations due December 31, 2020.
•	State Board adoption of potential new standards in early 2021.
•	Call for Social Studies instructional materials, including Civics, begins November 2021.

Slide 13
Slide 14
The Justice Sandra Day O’Connor Civics Education Act (CS/HB 105) was unanimously passed during the 2010 Florida Legislative Session, and then modified in 2013. In the legislation, changes were made to Section 1003.4156, Florida Statutes, that required three middle grades or higher courses in social studies, and that one of these courses must be at least a one-semester civics education course. 
Beginning with the 2013–2014 academic year, students enrolled in relevant courses were required to take the Civics EOC Assessment, and the results of the EOC were required to comprise 30% of the final course grade. 
Slide 15
Listed here for your reference are the various Civics courses in which enrolled students would be required to take the Civics EOC. I won’t read the extensive list of course names and course numbers now, but the list will be included in the ADA-compliant version of the presentation posted online, and it will also be listed in the transcript that will accompany this recorded webinar. 
[Students enrolled in the following courses are required to participate in the Civics EOC Assessment (course number follows course name):
M/J Civics – 2106010
M/J Civics – 2106015
M/J Civics & Career Planning – 2106016
M/J Civics, Advanced – 2106020
M/J Civics, Advanced – 2106025
M/J Civics, Advanced & Career Planning – 2106026
M/J Civics and Digital Technologies – 2106029
M/J U.S. History & Civics – 2100045
M/J IB Middle Years Program Civics Advanced – 2106027
M/J IB Middle Years Program Civics Advanced & Career Planning – 2106028]
Slide 16
From 2010-2013, content of the Civics EOC was developed based on Florida course descriptions and recommendations of Florida content experts including university faculty, K-12 district content experts, K-12 educators, and Florida stakeholders. I’ll speak a bit more about the test development process in a few moments.
In 2012–2013, a statewide Civics EOC field test was administered to gather data on newly-developed test items, which helped determine which of the items would be used on future tests to measure student mastery of the course content. The following year, in 2013–2014, the baseline Civics EOC was administered to gather data to help determine student performance expectations on all future Civics EOC administrations. 
In summer 2014, a Standard Setting panel was convened. The panel, consisting of 26 Florida K-12 district content experts and K-12 educators, made recommendations regarding achievement standards for each Achievement Level (Levels 1-5, with Level 3 considered “passing,” per state statute).
Slide 17
Immediately following the Standard Setting panel, an 18-member Reactor Panel was convened to review the recommendations. This panel was comprised of Florida business leaders, representatives of parent groups, district superintendents, university and college leadership, and Florida expert stakeholder groups.
Using the recommendations from the two panels, and after considering other public input gathered during the State Board Rule notice period, the Commissioner made final recommendations that were established in Rule 6A-1.09422, Florida Administrative Code (F.A.C.), by the State Board of Education in January 2015.
More detail on the entire Standard Setting process is available on the Department’s Standard Setting webpage
Slide 18
The intent of the Specifications review is to gather recommendations regarding changes that may be desired as to which standards/benchmarks are assessed, and how to assess them on the statewide Civics EOC. 
Using the final, approved recommendations, the Specifications would be updated accordingly, and a systematic review would then be conducted for all existing Civics EOC test items to determine whether each still meets the Specifications. 
Items that do not meet the new Specifications may be removed entirely from eligibility for use on future test forms; or they may be edited as appropriate, reviewed by educators and content experts, and then re-field-tested for future use.
Slide 19
Moving forward, all new Civics test item development would adhere to the new Specifications. 
Changes regarding the overall structure of the test, such as the broad reporting categories under which the benchmarks are currently placed, may also result from this public review. 
In summary: the review of the Specifications is intended to provide stakeholders with the most thorough, thoughtful, and systematic way to address the final, approved recommendations for any changes to Civics EOC test items, as well as potential changes to the structure of the EOC. Reviews of individual test items, which will be discussed later in this webinar, are conducted annually by panels of Florida educators and content experts, and are not included in the Specifications review process.
Slide 20
The Civics EOC is delivered via a computer-based test platform
A small number of paper-based versions (regular print, large print, braille and one-item-per-page) are provided specifically for students with an Individual Educational Plan or Section 504 plan that calls for such an accommodation.
The Civics EOC is administered in one 160-minute session with a 10-minute break after the first 80 minutes; however, any student not finished by the end of the 160-minute session may continue working for up to the length of a typical school day.
Computer-based practice tests are publicly available, and students are required to participate in a practice session at their school prior to testing.
Paper-based regular-print practice tests are also publicly available.
Slide 21
For the next portion of this presentation, I’ll talk briefly about the purpose and some of the key content of the Specifications. The link to the publicly-available Specifications is at the bottom of the slide, and I encourage you to familiarize yourself with the document before and during your review. The first four sections of the document – the introduction; criteria for test items; item difficulty and cognitive complexity; and review procedures for test items – are not specifically addressed in the review process, but feedback is welcome. I will refer to some aspects of these sections later in the webinar, and you may refer to them at your convenience for more information. http://www.fldoe.org/accountability/assessments/k-12-student-assessment/end-of-course-eoc-assessments/ 
Slide 22
There are three general phases of the test development process that lead up to the phase in which Florida’s statewide assessments are administered. The Specifications are relied on most heavily during the first two phases, which are item development and item review. Test item writers are trained by FDOE and contractor staff to draft test items according to the Specifications. Department staff review the items and may suggest edits as appropriate before all test items are then reviewed by various committees of Florida educators, postsecondary content experts, and community stakeholders. Later in this presentation, you’ll see the types of things content reviewers are looking for during this process. Test items are also reviewed to ensure that there is no bias in favor or against any individuals or groups, and that test items are free from issues that may be sensitive to Florida stakeholders in general. These review processes are summarized on p. 14 of the Specifications. All test items also go through editorial reviews to ensure proper formatting, and to ensure that correct spelling, punctuation, and grammar is used. 
Each summer, immediately after scores are reported for the previous spring, test data is reviewed and test items are selected for the following spring’s tests. FDOE and contractor staff work together to create test forms, which are each reviewed and given final approval by leadership in the Office of Assessment. For Civics, an additional review by Florida content experts then takes place to ensure that any recent legislative changes have not impacted the accuracy of any of the test items on the forms approved for use the following spring.  
Slide 23
Page 16 of the Specifications provides definitions of important terms used to provide specific information for each individual benchmark specification within the document. It will be helpful for your review to first familiarize yourself with these definitions, and to refer back to them as needed throughout the review process. 
Slide 24
Using an example of one of the assessed benchmarks, p. 20 of the Specifications provides specific information regarding Benchmark SS.7.C.1.2. The detail provided here and for all other assessed benchmarks supports accurate item writing and reviewing.
Item writers and reviewers use the benchmark to identify WHAT the Civics EOC assesses. The Reporting Category and Standard show WHY the identified content is assessed by the Civics EOC; that is, “How does it fit into the overarching structure and purpose of the test?” And the four other components, which are the  Benchmark Clarifications, Content Limit, Stimulus Attribute, and Content Focus entries, guide item writers and reviewers to ensure that the benchmark is assessed within the parameters set forth in the Specifications.
Slide 25
Each Individual Benchmark Specification is immediately followed by a sample test item that illustrates for item writers, test item reviewers, and interested stakeholders one way in which the benchmark might be appropriately assessed. Here we see the sample item for the Benchmark that was shown on the previous slide.
Slide 26
The Specifications document includes five Appendices, which are listed here. Appendix A sets forth item writing and reviewing criteria, which will be discussed on the following slide to illustrate specifically what test item reviewers are asked to examine. Other appendices in the Test Item Specifications offer: information about the standards-based curriculum content assessed by the Civics EOC; an Item Writer Glossary; a test design summary; and Civics course descriptions. Specifications reviewers will be asked specifically for feedback on Appendix B and C, while feedback provided on the Individual Benchmark Specifications will help inform potential changes to the test design and therefore to Appendix D. The content of Appendix E is left for a separate review process, as Mike DiPierro discussed earlier.
Regarding the glossary, it is important to note that it is ONLY a reference list provided for the item writers and item reviewers, and is NOT intended to comprise a comprehensive vocabulary list for students.
I will also take this opportunity to note that the Content Focus terms presented in Individual Item Specifications are given in addition to those found in the standards, benchmarks, and benchmark clarifications. Content Focus terms may include, but are not limited to, the terms listed. Content focus data is not intended for use as an indicator of performance of students or groups of students, rather as an additional guide for item writers and test item reviewers.
Slide 27
The item writing and item reviewing criteria provided in Appendix A of the Specifications includes a series of questions used by members of the Civics EOC Content Expert and Item Review Committee to evaluate each proposed test item. Decisions whether to accept, revise, or reject any particular test item are based largely upon the answers to these questions. 
Slide 28
The test design summary shown in Appendix D illustrates the four broad Reporting Categories that the Benchmarks are currently distributed among. Each of the Reporting Categories currently comprises 25% of the total test. As noted, feedback solicited elsewhere in the review will help inform potential changes to the test design and to this particular appendix. 
Slide 29
In closing, I’ll spend a few moments discussing the Specifications review process itself. As you proceed through the review, you will see key questions associated with each Individual Benchmark Specification, as well as questions regarding other aspects of the Specifications. Clear and thorough answers to these questions will help guide recommendations for revisions. Reviewers are asked to offer recommendations that are as specific as possible to help ensure that any approved revisions are implemented in the way that they were intended.  
Slide 30
As with the Instructional Materials review, stakeholders may provide feedback on the Specifications via the EdCredible® platform, accessed via the link provided here: www.floridacivicsreview.org.
As I mentioned earlier, the feedback window opened Thursday, August 15th, and will remain open through Tuesday, October 15th. As a reminder, feedback regarding Civics standards, benchmarks, course descriptions, and instructional materials is being gathered, or will be gathered through separate initiatives. 
Accordingly, for this specific aspect of the review, stakeholders will be asked to limit feedback to the Civics EOC Test Item Specifications.
Slide 31
For information on how to use the EdCredible® platform to participate in the Civics instructional materials or Specifications review, please watch the video tutorials posted to www.floridacivicsreview.org.
Slide 32
As with the Instructional Materials review, after the public review period, comments will be compiled for the Commissioner’s review in order for him to make any recommendations for improvements to the Specifications by December 31, 2019, as required in HB 807.
Updates about the Civics instructional materials and Specifications reviews, including the recording, presentation, and transcript of this webinar, will be posted at http://www.fldoe.org/civicsreview.
Slide 33
Now we will open the webinar to questions, and answer those that we’ve received during the presentation. As a reminder, you may use the webinar chat feature available during the live presentation, or email questions regarding either the Instructional Materials review or the Specifications review to CivicsReview@fldoe.org. Mike or I will respond to chat questions here as appropriate. Answers to these questions, and answers to other questions that need to first be researched further will be posted to the Civics Review page, along with the presentation you are viewing, the webinar recording, and the transcript of this webinar.