FCAT Writing+
Scoring/Reporting Issues
2007 Annual Assessment Coordinators Meeting

“Accept the challenges, so that you may feel the exhilaration of victory.” General George S. Patton

---

Topics

• Timeline for Writing+

• Composite Scores Reported

• 2007 unscorables

• 2008 Grade 10 Change

---

FCAT Writing+

Measures the Sunshine State Standards in Writing

1993: State test of writing skills using essay

1999: Tenth Grade Writing Graduation Requirement in Law

2005: Multiple-choice items added to essay

2006: Scores reported on Writing+ (plus)

2007: Report Achievement Levels

2008: Grade 10 students receive scores that count for graduation

2010: First class with Writing+ (plus) Graduation requirement

---

The Essay Portion

• “On demand” draft writing in a 45 minute time period

• No knowledge of topic in advance

• Planning sheet and two lined pages for draft

• Grade 4: Narrative and Expository Modes

• Grade 8 and 10: Expository and Persuasive Modes

• Scored by trained readers on a scale of 1 to 6

---

FCAT Writing+ Scores

• Total scale score between 100 and 500

  • • Essay score (a rubric score of 1 to 6)

    • Multiple-choice scores for four elements of writing (points correct out of points possible)

      • Focus

      • Organization

      • Support

      • Conventions

• Total scale score is a Composite Score (Essay score + Multiple-Choice score)

---

FCAT Writing+ Composite Scores

---

2007 Writing Issue

• The number of unscorable expository essays doubled

• Examination of the Issue

  • Reviewed historical data

  • Reviewed 2007 process

  • Reviewed 2007 papers

---

Scoring Process

• Answer documents are scanned

• Images created and displayed on computer monitors – Several enhancements over paper

  • enlargement, brightness, & contrast

• Process of Assigning Unscorable Codes

  • Scorers can assign only Blanks=A

  • Supervisors and DOE staff on site assign other unscorable codes

  • Unclear papers can be pulled

• No change in the process from previous years

---

What is Unscorable?

• Do not score codes – local decisions

• Blank documents (Code=A)

• Unscorable Ratings (assigned only by DOE or scoring supervisor)

  • Off topic (Code=B)

  • Illegible, Incomprehensible, or Insufficient (Code=C)

  • Foreign language (Code D)

---

2007 Increase in Unscorables

• Grade 4

  • Primary Reason: Illegible

  • Increase over 2006: Illegible on Expository

• Grade 8

  • Primary Reason: Illegible

  • Increase over 2006: Off Topic on Expository

• Grade 10

  • Primary Reason: Illegible

  • Increase over 2006: Illegible on Expository

---

Comparable Data – Other Years

• Grade 4 – 0.46% to 1.01%

  • 2002 Exp=1.01%; 2002 Nar.=0.94%

  • 2007: Exp=0.91%; Nar.=0.49%

• Grade 8 – 0.22% to 54%

  • 2004 Exp=0.52%; 2002 Per=0.54%

  • 2007: Exp=0.54%; Per=0.46%

• Grade 10 – 0.16% to 0.80%

  • 2005 Exp=0.53%; 2000 Per=0.56%

  • 2007: Exp=0.80%; Per=0.77%

---

Grade 10 Spring 2008 Change

• Only 1 prompt

• Randomly selected mode

  • Expository

  • Persuasive

• Retakes – only one prompt

• No change for grade 4 and 8

---

Assessment Issues
2007 Annual Assessment Coordinators Meeting

“Accept the challenges, so that you may feel the exhilaration of victory.” General George S. Patton

---

Topics

• Grade 3 Scoring Anomaly

  • What happened?

• FCAT External Review

• Budget Cuts

---

The Grade 3 Test in 2006

• Passages – Questions – Forms

• Student scores based on 5 passages & 45 questions

• 30 different forms, each with 1 passage & 7-8 questions

• Forms are used for anchor and field test questions

• One of the 6 passage positions is used for anchor and field test questions

---

The Grade 3 Test in 2006

---

“Test Construction”

• Process of building the test

  • Occurs the summer before a test

  • Based on available passages, questions, and statistics

• Guidelines for building the test

  • Test Construction Specifications

• Building the test is an iterative process

---

Equating Solutions Issues

• The 2006 equating solution was affected by location changes of the anchor questions

  • Some change in question location is expected

  • In 2006, a large number of questions changed locations

  • 12/21 anchor questions moved more than 15 item positions (0 in 2004 and 2005, and 2 in 2007)

  • The result was better performance

  • However, no outliers were identified via calibration

• Goal is to find a “better” or “best” equating solution

  • “Better” must be defined

  • Process considerations

    • Selection of anchor questions

    • Follow the guidelines

    • Evaluate the quality of the anchor questions

---

FCAT External Review Process

• Advisory Committee

  • Superintendents and Assessment Coordinators

  • External Groups – FCAR, Unions, Business Representative

• Ad hoc technical subcommittee

  • Identify and select Independent Reviewers

  • Identify other technical topics for the full committee

  • Next Meetings

    • Thursday, 5:30

    • October 2 – Royal Caribe Hotel, Orlando

---

Independent Review Contract

• Review of Experts (individuals and organizations)

• Request for Application (non-profits or university affiliates)

• Two respondents – Buros Institute (selected) and EduMetrics

• Scope of Work

  • Independent review of 2008 tests

  • Review of Florida equating procedures

  • Review/Recommendations about 2006 Grade 3 analyses

  • Concurrent scoring oversight

  • Other issues - TBD

---

Changes Proposed for 2008

• Strict limits on passage/item movements

• P-value matched between scored test and anchor items

• All equating completed before any grade level is reported

  • Grade 3 results delayed

• Concurrent quality control of equating by independent reviewers

  • May delay reporting of all grades

---

Budget Cuts – Impacts on Assessment

• Meeting Participants

  • Increase in refusals – HELP!

  • Support for substitutes in Federal $ for professional development

• Other Scenarios

  • Reduce printed products

  • Stop the FCAT Parent Network

  • Delay the NRT implementation timeline

  • Writing+ graduation requirement

  • Reduce FTCE administration and revision