Comprehensive English Language Learning Assessment (CELLA)

TRAIN THE TRAINER

TRAINING SESSIONS -- January 2009

Speaker Notes:

---

Welcome & Introductions

• Lori Rodriguez, Chief, Bureau of Academic Achievement through Language Acquisition

• Wendy Nucci, ETS CELLA Program Manager

• Paul Rybinski, ETS Assessment Development Lead, English Language Learning

• Zulma Torres, ETS CELLA Program Manager

• Lorenzo Murolo, Inbound Operations Manager

Speaker Notes:

---

Morning Agenda

• Welcome & Introductions

• Review Content of Tests

• Who Should be Tested

• Functional Level Testing

  • What is it?

  • How is it administered?

• Administering Level A

• Administering Levels B-D

Speaker Notes:

---

Afternoon Agenda

• What’s New for 2009

• 2009 Comprehensive Schedule

• Training Materials

• 2009 Supplemental Order Process

• Responsibilities

  • Test Administrator

  • School Coordinator

  • District Coordinator

• 2009 DFAs and Scoring Guides

• Answer Docs and Pre-ID Labels

• Question and Answers Session

Speaker Notes:

---

What is CELLA?

CELLA provides…

• evidence of program accountability in accordance to Title III of NCLB

• data on students over time

• information about the language proficiency of students across the State

• diagnostic information about students’ strength and weakness in English

• fair, reliable, and valid information about ELLs in grades K through 12

Speaker Notes:

---

How was CELLA developed?

• CELLA was funded by a grant from the U.S. Department of Education.

• A collaboration with AccountabilityWorks and a consortium of five states: Florida, Maryland, Michigan, Pennsylvania, & Tennessee

• As the grantee, AccountabilityWorks managed the work of the consortium

• ETS was awarded the subcontract to develop the test, ancillary materials, conduct the field test and develop the scoring programs

• The State provided substantial input and the majority of field tested students

Speaker Notes:

---

CELLA
Test Content

Speaker Notes:

---

Overview of CELLA

Test Sections:

• • Listening

  • Speaking

  • Reading

  • Writing

Test Levels:

• • Level A (Grades K-2)

  • Level B (Grades 3-5)

  • Level C (Grades 6-8)

  • Level D (Grades 9-12)

Test Format:

• • Listening: All Multiple Choice

  • Speaking: All Constructed Response

  • Reading: All Multiple Choice

  • Writing: Multiple Choice & Constructed Response

Speaker Notes:

Facilitator Notes:

Review slide; highlight important points/concepts.

The listening and speaking sections of CELLA are administered at grade level. The reading and writing sections may be assessed at grade level or below grade level. Explain that CELLA allows for functional level testing and that this will be explained in more detail in subsequent slides.

---

CELLA Item Types

Listening

• Listen & Match

• Picture Description

• Short Talks

• Extended Listening

Reading Vocabulary

• Synonym

• Antonym

• Idiom

• Root & Affix

Writing Multiple Choice

• Synonym

• Grammar, structure
  & written expression

• Paragraph choices

• Recognizing errors

Speaking

• Oral Vocabulary

• Speech Functions

• Personal Opinion

• Story Retelling

• Graph Interpretation

Reading Comprehension

• Main Idea

• Detail

• Inference/Prediction

• Reference

• Rhetorical Elements

• Vocabulary in Context

Constructed Response

• Writing sentences

• Writing paragraphs

Speaker Notes:

Facilitator Notes:

The listening test is all multiple-choice and is paced by a CD (or teacher script, which appears in the Directions for Administration manual).

There are four item types that progressively get more challenging.

1. Listen and Match: single sentences—taps into the lower end of the proficiency spectrum.

2. Picture Description: single sentences, but more listening is involved.

3. Short Talks: two types—partial dialogues and teacher talks (mixture of BICS and CALP).

BICS – Basic Interpersonal Communication Skills or the language skills needed in social situations.

CALP – Cognitive Academic Language Proficiency or formal academic learning. This includes listening, speaking, reading, and writing about subject area content material.

4. Extended Listening: assesses whether the student can learn new information in English.

The Speaking section is administered in a one-on-one setting. The test administrator sits with a student and asks a series of questions. The student’s responses are scored on the spot by the test administrator.

There are 5 items types in the speaking tests:

1. Oral Vocabulary – student identifies objects or actions and state antonyms

2. Speech Functions – student asks a question related to a situation

3. Personal Opinion – student gives reason to support an opinion

4. Story Retelling – student hears story then repeats it

5. Graph Interpretation – student looks at graph and speaks about the information on the graph

---

Test Document Configuration
(On Grade Level – ALL Sections)

Images are provided for the following test documents: Level A: Grade K – 2, Level B: Grades 3 – 5, Level C: Grades 6 – 8, Level D: Grades 9 – 12.

Speaker Notes:

---

Testing Decision
Tree
(Whom Do
You Test?)

Speaker Notes:

---

Which Students Should be Tested?

• All students in grades K-12 classified as ELLs, with codes of “LY” or “LP” as of May 31, 2008 and in membership in the district at the time of testing

• Former ELLs (code of “LF”) that were exited from the ESOL program on or after May 31, 2008 and before April 14, 2009

• Any student that enrolls after January 16, 2009 and before April 3, 2009, and is classified as an ELL must be assessed

Speaker Notes:

Facilitator Notes:

Review slide; highlight important points/concepts.

Reminder: Facilitator should note that CELLA is also aligned to the Language Arts through ESOL Performance Indicators and is linked to the Sunshine State Standards for Language Arts. CELLA is designed to test not only language proficiency skills in all four domains of the English language, but also the use of academic English.

---

Who is NOT Tested?

• Students enrolling on or after April 6, 2009, with affirmative responses on the Home Language Survey are NOT included in the 2009 administration of CELLA

Speaker Notes:

---

Functional
Level Testing

Speaker Notes:

---

What is Functional Level Testing?

• Testing students’ language ability according to what they can actually do and NOT according to what is expected at their grade level

• Used only for literacy skills (Reading & Writing), which must be learned, and NOT for oral skills (Listening & Speaking), which are more naturally “acquired”

• Students may take the Reading and Writing sections at a level lower than grade level

Speaker Notes:

---

• There is a often wide disparity in skill levels of ELLs (especially in the higher grades)

• To get reliable and accurate test results for students at all proficiency levels within each grade-span would require more test items

• Test administration time is kept short by giving students only those items most appropriate for their skill level

• Without functional level testing, measures of growth are inaccurate for very low performing students

Why administer Functional Level Testing?

Speaker Notes:

• The widest disparity in skill levels is typically found in the high school grades, where ELLs can range from recently arrived, non-English speaking students with no native-language literacy skills to high-performing students ready to demonstrate sufficient English language proficiency to exit the English for Speakers of Other Languages (ESOL) program.

• In order to provide precise and reliable results for each student at each CELLA level, additional test items would have to be administered to ensure that all proficiency levels are tested within each grade-span.

• To keep testing time as short as possible, CELLA was designed to assess students according to their "functional" skill levels in Reading and Writing rather than their actual grade level.

---

• Frustration during testing is minimized.

• Abilities are more accurately reflected by the scores.

• Interventions are more effectively and appropriately.

• Results can be compared across districts more accurately.

Benefits of Functional Level Testing
to the Student

Speaker Notes:

• Using the full range of Reading and Writing levels is the best way to ensure reliable results for all ELLs as well as to avoid student frustration during testing.

• Administering the most appropriate level of Reading and Writing ensures that each student is given only test items that reflect his or her skill level, resulting in a highly accurate score on the CELLA vertical scale and providing pertinent diagnostic information.

• Administering the most appropriate level of Reading and Writing ensures accurate identification and implementation of interventions for students.

• Uniform implementation of functional-level testing throughout a district provides accurate and reliable results of assessment of ELLs and ensures fairness when comparing student results across schools.

• Uniform implementation of functional level testing across the state ensures the fair and accurate application of sanctions by FDOE to districts whose ELL population fails to demonstrate progress in meeting Florida’s Annual Measurable Achievement Objectives (AMAOs) for two or more consecutive years.

---

CELLA Results

• Administering the CELLA Reading & Writing sections according to students' functional level does NOT result in an "off-grade" version of the assessment

• “Off-grade" testing typically indicates that student results are scored and reported against a different, lower standard. This is NOT the case with CELLA

• Every student in a given grade span (e.g., K-2, 3-5, 6-8, 9-12), regardless of which test level is administered, will be reported as "English Proficient" against the same standard (or "cut-score") on the assessment's vertical scale

Speaker Notes:

---

How To Implement CELLA
Functional Level Testing

Teachers and CELLA Test Administrators should use the following tools to determine the correct Reading & Writing level to administer to students classified as ELLs at the time of testing:

2009 CELLA Testing Decision Flowchart, AND

2009 CELLA Functional Level Testing Decision- Making Rubric,

-OR-

CELLA Online Locator Test

Speaker Notes:

Facilitator Notes:

Review slide; highlight important points/concepts.

Reminder: Facilitator should note that CELLA is also aligned to the Language Arts through ESOL Performance Indicators and is linked to the Sunshine State Standards for Language Arts. CELLA is designed to test not only language proficiency skills in all four domains of the English language, but also the use of academic English.

---

Which students should NOT
be tested with
Functional Level Testing?

Former ELLs (LFs) who exited the ESOL program on or after May 31, 2008 must be tested in Reading & Writing based on their grade level

Speaker Notes:

---

2009 CELLA Functional Level Testing Decision-Making Rubric

• Developed based on the CELLA test items and the content demands in Reading & Writing required of students to reliably determine their level of English proficiency in these two domains

  • Teachers use a check mark next to the level of the student’s attainment in each of the four English skill areas

  • Students with 3 or 4 check marks in one column should take the test indicated in that column

Speaker Notes:

---

2009 CELLA FUNCTIONAL LEVEL TESTING
DECISION-MAKING RUBRIC

Speaker Notes:

---

How to Administer Functional
Level Testing

Speaker Notes:

---

Test Material for Functional
Level Testing

Speaker Notes:

Facilitator Notes:

Walk through what testing material is used for students who are to be tested using functional level testing – Listening and Speaking on grade level and Reading and Writing at a lower level.

---

FUNCTIONAL LEVEL TESTING – Example 1

Example: A student in grade 9 will take Level D, Listening and Speaking

The student is eligible to take the Level C, Level B, or Level A Reading & Writing test (images of each test document are provided below)

ALL student responses are recorded onto the Level B, C, D Answer Document

ALL Student responses are recorded on the Functional Level A Test Book

Speaker Notes:

Facilitator Notes:

A student in grade 9 will take Level D, Listening and Speaking and the student is eligible to take either the Level C or Level B or Level A Reading and Writing test.

If the student takes the Level C or Level B Reading and Writing, the student will record all responses in the 2008 CELLA Level B, C and D Answer Sheet.

If the student takes the Level A Reading and Writing, the student will record ALL responses in the Functional Level A Test Book.

---

FUNCTIONAL LEVEL TESTING – Example 2

Example: A student in grade 5 will take Level B Listening & Speaking and is eligible to take the Level A Reading & Writing test (images of each test document are provided below)

ALL student responses are recorded onto

the Functional Level A Test Book

Speaker Notes:

Facilitator Notes:

Review slide; highlight important points/concepts.

A student in grade 5 will take Level B, Listening and Speaking B and is eligible to take the Level A Reading and Writing test. All of the student’s responses are recorded in the Level A Reading and writing test book.

This student would NOT use a 2008 CELLA Level B, C, D Answer Document.

---

Level A
Training

Speaker Notes:

---

Level A Sections

Speaker Notes:

---

Administering Level A

• The test should be individually administered to students in kindergarten.

• Grades 1 and 2 may take Listening, Reading and Writing in small groups.

• The One-on-One section must be individually administered

• Grade 2 will take additional “extension” items in Reading & Writing

Speaker Notes:

---

Administering Level A
“Core” and “Extension”

• The Core Section should be taken by all students in Grades K-2

  • Listening & Speaking sections are identical for all students in grades K-2

• The Extension contains more challenging Reading & Writing items and is administered only to students in Grade 2

  • Students in grades K and 1 receive fewer Reading & Writing items than students in Grade 2

Speaker Notes:

The Reading and Writing tests have clearly marked stopping points in the DFA for students in K and 1. In addition, since the test is administered item-by-item, the administrator will inform students when it is time to stop. Students in grade 2 will be directed by the administrator to continue past those stopping points.

---

Administering Level A

STOPPING POINTS are available for individually administered sections of CELLA:

• • Test administrators should stop the administration of that section only IF the student cannot respond to five questions in a row AND it is clear that the questions are above the student’s proficiency level

Speaker Notes:

Facilitator Notes:

Students taking CELLA may have a wide range of English proficiency levels. Some students will easily answer all of the questions; others may be unable to answer even the easiest of questions. The goal of providing stopping points is to give students an opportunity to demonstrate as much proficiency as they have, without overtaxing them by asking them questions that they are clearly not able to answer.

NOTE: administration of the section should be stopped ONLY if the student cannot respond to questions; if a student is engaged and answering questions but gets five (5) questions in a row wrong, the administration should continue.

After administration of a section is stopped for the above outlined reasons, the student should continue to attempt the other sections of CELLA at the regularly scheduled time.

---

Level A Student Responses

Please demonstrate how students should mark their responses in their test book:

Correct

Incorrect

Speaker Notes:

---

Test Administrators
One-on-One Scoring Section

Page 38 of the Level A Student Test Book

Speaker Notes:

---

CELLA Speaking/One-on-One Administration
Knowledge & Skills Needed to be an Effective Evaluator of Language Proficiency

• • Proficiency in English

  • Thorough understanding of how rubrics “work”

  • Knowledge of and experience with the specific skills being measured

  • Practice with rubrics

  • Objectivity

Speaker Notes:

Facilitator Notes:

Personnel responsible for administering CELLA must be proficient in listening, speaking, reading and writing English (Sufficient proficiency in English to effectively apply the scoring rubric and evaluate students’ responses in English. The proficiency of Test Administrators/Scorers should be determined by the District Coordinator for CELLA), and must have completed training for the appropriate CELLA Speaking test level and/or the One-on-One section.

---

Administering Level A

One-On-One Section
Speech Functions

Measures a student’s oral response to a specific prompt

Criteria include:

• • Appropriateness of information

  • Grammatical accuracy

Speaker Notes:

Facilitator Notes:

1. Read and review with participants the Rubric Review Activity Directions (handout #2) found in participant handout packet for Level A.

2. Conduct the activity. Participants will use the rubric from the Scoring Guide, Speech Functions for this activity.

3. Tab with a Post-it for quick reference in the next activity.

---

Administering Level A

One-On-One Section Speech Functions

Rubric Review Activity

• There are a total of 22 training tracks for Speech Functions

• Refer to the Directions for Administration &Scoring Guides

  • Rubric is on page 49

  • Training Items are on page 56

  • Answer Key is on page 63

Activity

1. 1. 1. Listen to training CD Tracks 1-2

      2. Discuss why the response scored a 2, referring to the rubric

      3. Listen to CD Tracks 3-4

      4. Discuss why the response scored a 1 and 0

      5. Make sure you LISTEN to student responses, rather than read the transcript

Speaker Notes:

---

Administering Level A

One-On-One Section
Personal Opinion

Measures students ability to orally state and defend an opinion

Criteria include:

• • Clarity of response

  • Adequate support

  • Good control of grammar & adequate vocabulary

Speaker Notes:

Facilitator Notes:

Review Slide

---

Administering Level A

One-On-One Section Personal Opinion

Rubric Review Activity

• There are a total of 9 training tracks for Personal Opinion

• Refer to the Directions for Administration & Scoring Guides

  • Rubric is on page 50

  • Training Items are on page 57

  • Answer Key is on page 64

Activity

1. 1. 1. Listen to training CD Tracks 23-24

      2. Discuss why the response scored a 2, referring to the rubric

      3. Listen to CD Track 27

      4. Discuss why the response scored a 1

      5. Make sure you LISTEN to student responses, rather than read the transcript

Speaker Notes:

---

Administering Level A

One-On-One Section
Story Retelling

Measures a student’s ability to hear a story (while looking at sequential picture cues) and to then retell it with detail

Criteria include:

• • • • Comprehensive response

      • Vocabulary

      • Grammar

      • Fluency

Speaker Notes:

Facilitator Notes:

Review Slide

---

Administering Level A

One-On-One Section Story Retelling

Rubric Review Activity

• There are a total of 16 training tracks for Story Retelling

• Refer to the Directions for Administration & Scoring Guides

  • Rubric is on page 51

  • Training Items are on page 58

  • Answer Key is on page 64

Activity

1. 1. 1. Listen to training CD Tracks 32-33

      2. Discuss why the response scored a 4, referring to the rubric

      3. Listen to CD Track 35

      4. Discuss why the response scored a 2

      5. Make sure you LISTEN to student responses, rather than read the transcript

Speaker Notes:

---

Administering Level A

One-On-One Section
Reading Aloud for Fluency

Measures reading fluency

Criteria include:

• • • • • Rate

        • Accuracy

Speaker Notes:

Facilitator Notes:

Review Slide

---

Administering Level A

One-On-One Section
Reading Aloud for Fluency

Administration - Types of Errors:

• • Substitution

    • e.g., bird instead of bear

  • Mispronunciation

    • e.g., fell instead of fall

    • Words pronounced with an accent are counted as correct if they cannot be confused with other English words

  • Omissions

    • i.e., skipped words

    • If the student stops or struggles with a word for 3 seconds, you may tell the student the word and count it as an error

Speaker Notes:

Facilitator Notes:

In reviewing this slide, make sure that participants understand that there are only 3 error indicators; substitution, mispronunciations, and omissions. They do not have to indicate the type of error, simply record the total number of errors.

Sometimes participants want to compare this scoring method to the Running Record. State that this is not a running record and repeat the types of errors to count.

See page 16 of the Scoring Guide for more information regarding errors.

---

Administering Level A

One-On-One Section
Reading Aloud for Fluency

There are a total of 16 training tracks for Reading Aloud for Fluency

• Refer to the Directions for Administration & Scoring Guides

  • Rubric is on page 53

  • Training Items are on page 62

  • Answer Key is on page 66

Activity

1. 1. 1. Listen to training CD Tracks 48-49

      2. Discuss why the response scored a 4, referring to the rubric

      3. Listen to CD Track 53

      4. Discuss why the response scored a 0

      5. Make sure you LISTEN to student responses, rather than read the transcript

Note:

• • If the student makes repeated errors on the same word, count the error only once

  • Repetitions and self-corrections are not counted as errors

Speaker Notes:

Facilitator Notes:

Scoring Procedure:

1. To prepare for the practice, ask the participants to turn to the Level A Question 5: Read Aloud for Fluency in the DFA and Scoring Guide. Clip a blank acetate transparency over page 20 or insert the Scoring Guide into a clear sheet protector. The participants now have the transparency placed directly over the Fluency text.

2. As the student reads, participants follow along using text in DFA and Scoring Guide.

3. Participants mark the student errors on transparency as they occur by drawing a line through the text with dry erase pen. Participants need only mark the error, not record the type of error.

4. Participants mark where student stops the reading at end of allotted time by drawing a vertical line.

5. See rubric in DFA and Scoring Guide for scoring student response.

6. Clean transparency after each student.

Note: Model and practice this procedure during sample responses.

This is where you will practice using the rubric to score student responses. For this activity, participants will refer Training Items on the following pages in the DFA and Scoring Guide. It will be helpful to have participants tab the pages for quick reference.

1. Ask participants to review the Script for Training Items so they are familiar with the script format and content.

2. Ask participants to turn to back to the rubric.

3. Facilitator plays training CD Tracks 48-49.

4. Participants listen to Question 5 followed by student response, Score 4. (Tracks 48-49)

5. Facilitator pauses the CD and ask participants to discuss why that particular response scored a 4. Refer to rubric.

6. Repeat this process for scores 3, 2, 1 and 0, Tracks 50-53.

7. The facilitator will now play training item Example A , Track 54 and ask participants to practice scoring using the rubric and the training items.

8. After the participants practice scoring Example A, ask them to compare scores with a partner or table-mates.

9. Ask participants to turn to the Answer Key for Training Items. Facilitator shares ETS score and words counted as errors.

10. Invite discussion about items when necessary.

11. Repeat steps 7-10 for the following Example training items:

Example B, Track 55

Example C, Track 56

Example D, Track 57

Example E, Tracks 58

1. 12. For additional practice, continue steps 7-10 with Examples F-J, Tracks 59-63.

Note: Often after practice and scoring, participants raise questions about the “allowed” usage of probing questions and prompts. The facilitator will need to keep participants focused on the purpose of CELLA; to measure English Language Proficiency.

---

Level B-D
Listening/Speaking
for Grades 3-12

Speaker Notes:

---

Administering Levels B-D Listening & Speaking

Grades:

• • Level B (3-5)

  • Level C (6-8)

  • Level D (9-12)

• Test design is identical across levels for Listening & Speaking*

• Listening all multiple-choice; Speaking all constructed-response

• Listening is group administered; testing time is about 25 minutes

* Level B is the only exception with one additional item

Speaker Notes:

At Levels B through D, the Listening and Speaking sections are identical in design (with one minor exception in the Level B speaking test, which has one more item.) Thus, except for that one item, the process for administering these tests is identical.

All 22 Listening items are multiple-choice, and the test is administered to a group. The test is recorded on a CD. If there are technical problems, the entire script (including directions for how long to pause) is included in the DFA. The entire test is paced by the CD (or script).

The Speaking section contains 13 constructed-response items. The section is given to each student individually, so each school should create a schedule to assure that each student is tested.

---

CELLA B-D
Listening & Speaking Sections

Listening

Listen and Match

Picture Description

Short Talks

Extended Listening

Materials Needed

Levels B, C & D Answer Sheet

Directions for Administration & Scoring Guides

Listening & Speaking test book

Listening CD

Speaking

Oral Vocabulary

Speech Functions

Personal Opinion

Story Retelling

Graph Interpretation

Materials Needed

Levels B, C & D Answer Sheet

Directions for Administration & Scoring Guides

Listening & Speaking Test Book

Training CD for Speaking

Speaker Notes:

Facilitator Notes:

The listening test is all multiple-choice and is paced by a CD (or teacher script, which appears in the Directions for Administration manual).

There are four item types that progressively get more challenging.

1. Listen and Match: single sentences—taps into the lower end of the proficiency spectrum.

2. Picture Description: single sentences, but more listening is involved.

3. Short Talks: two types—partial dialogues and teacher talks (mixture of BICS and CALP).

BICS – Basic Interpersonal Communication Skills or the language skills needed in social situations.

CALP – Cognitive Academic Language Proficiency or formal academic learning. This includes listening, speaking, reading, and writing about subject area content material.

4. Extended Listening: assesses whether the student can learn new information in English.

The Speaking section is administered in a one-on-one setting. The test administrator sits with a student and asks a series of questions. The student’s responses are scored on the spot by the test administrator.

There are 5 items types in the speaking tests:

1. Oral Vocabulary – student identifies objects or actions and state antonyms

2. Speech Functions – student asks a question related to a situation

3. Personal Opinion – student gives reason to support an opinion

4. Story Retelling – student hears story then repeats it

5. Graph Interpretation – student looks at graph and speaks about the information on the graph

---

Test Level Selection Area

Speaker Notes:

---

Level C
Training

Speaker Notes:

---

Administering Level C
Speech Functions

Measures a student’s oral response to a specific prompt

Criteria include:

• • Appropriateness of information

  • Grammatical accuracy

Speaker Notes:

Facilitator Notes:

1. Read and review with participants the Rubric Review Activity Directions (handout #2) found in participant handout packet for Level A.

2. Conduct the activity. Participants will use the rubric from the Scoring Guide, Speech Functions for this activity.

3. Tab with a Post-it for quick reference in the next activity.

---

Administering Level C
Speech Functions
Rubric Review Activity

• There are a total of 45 training tracks for Speech Functions

• Refer to the Directions for Administration & Scoring Guides

  • Rubric is on page 186

  • Training Items are on page 194

  • Answer Key is on page 204

Activity

1. 1. 1. Listen to training CD Tracks 1-2

      2. Discuss why the response scored a 2, referring to the rubric

      3. Listen to CD Tracks 3-4

      4. Discuss why the response scored a 1 and 0

      5. Make sure you LISTEN to student responses, rather than read the transcript

Speaker Notes:

---

Administering Level C

Personal Opinion

Measures students ability to orally state and defend an opinion

Criteria include:

• • Clarity of response

  • Adequate support

  • Good control of grammar & adequate vocabulary

Speaker Notes:

Facilitator Notes:

Review Slide

---

Administering Level C
Personal Opinion
Rubric Review Activity

• There are a total of 13 training tracks for Personal Opinion

• Refer to the Directions for Administration & Scoring Guides

  • Rubric is on page 188

  • Training Items are on page 197

  • Answer Key is on page 206

Activity

1. 1. 1. Listen to training CD Tracks 46-47

      2. Discuss why the response scored a 2, referring to the rubric

      3. Listen to CD Track 48

      4. Discuss why the response scored a 1

      5. Make sure you LISTEN to student responses, rather than read the transcript

Speaker Notes:

---

Administering Level C

Story Retelling

Measures a student’s ability to hear a story (while looking at sequential picture cues) and to then retell it with detail

Criteria include:

• • • • Comprehensive response

      • Vocabulary

      • Grammar

      • Fluency

Speaker Notes:

Facilitator Notes:

Review Slide

---

Administering Level C
Story Retelling
Rubric Review Activity

• There are a total of 16 training tracks for Story Retelling

• Refer to the Directions for Administration & Scoring Guides

• Rubric is on page 198

  • Training Items are on page 200

  • Answer Key is on page 206

Activity

1. 1. 1. Listen to training CD Tracks 59-60

      2. Discuss why the response scored a 4, referring to the rubric

      3. Listen to CD Track 62

      4. Discuss why the response scored a 2

      5. Make sure you LISTEN to student responses, rather than read the transcript

Speaker Notes:

---

Administering Level C

Graph Interpretation

Measures students ability to orally summarize and interpret a graph

Criteria include:

• • • Summary response

    • Comparison response

    • Vocabulary

    • Grammar

    • Fluency

Speaker Notes:

Facilitator Notes:

Review Slide

---

Administering Level C
Graph Interpretation

• There are a total of 12 training tracks for Graph Interpretation

• Refer to the Directions for Administration & Scoring Guides

  • Rubric is on page 201

  • Training Items are on page 203

  • Answer Key is on page 208

Activity

1. 1. 1. Listen to training CD Tracks 75-76

      2. Discuss why the response scored a 4, referring to the rubric

      3. Listen to CD Track 80

      4. Discuss why the response scored a 0

      5. Make sure you LISTEN to student responses, rather than read the transcript

Speaker Notes:

Facilitator Notes:

Scoring Procedure:

1. To prepare for the practice, ask the participants to turn to the Level A Question 5: Read Aloud for Fluency in the DFA and Scoring Guide. Clip a blank acetate transparency over page 20 or insert the Scoring Guide into a clear sheet protector. The participants now have the transparency placed directly over the Fluency text.

2. As the student reads, participants follow along using text in DFA and Scoring Guide.

3. Participants mark the student errors on transparency as they occur by drawing a line through the text with dry erase pen. Participants need only mark the error, not record the type of error.

4. Participants mark where student stops the reading at end of allotted time by drawing a vertical line.

5. See rubric in DFA and Scoring Guide for scoring student response.

6. Clean transparency after each student.

Note: Model and practice this procedure during sample responses.

This is where you will practice using the rubric to score student responses. For this activity, participants will refer Training Items on the following pages in the DFA and Scoring Guide. It will be helpful to have participants tab the pages for quick reference.

1. Ask participants to review the Script for Training Items so they are familiar with the script format and content.

2. Ask participants to turn to back to the rubric.

3. Facilitator plays training CD Tracks 48-49.

4. Participants listen to Question 5 followed by student response, Score 4. (Tracks 48-49)

5. Facilitator pauses the CD and ask participants to discuss why that particular response scored a 4. Refer to rubric.

6. Repeat this process for scores 3, 2, 1 and 0, Tracks 50-53.

7. The facilitator will now play training item Example A , Track 54 and ask participants to practice scoring using the rubric and the training items.

8. After the participants practice scoring Example A, ask them to compare scores with a partner or table-mates.

9. Ask participants to turn to the Answer Key for Training Items. Facilitator shares ETS score and words counted as errors.

10. Invite discussion about items when necessary.

11. Repeat steps 7-10 for the following Example training items:

Example B, Track 55

Example C, Track 56

Example D, Track 57

Example E, Tracks 58

1. 12. For additional practice, continue steps 7-10 with Examples F-J, Tracks 59-63.

Note: Often after practice and scoring, participants raise questions about the “allowed” usage of probing questions and prompts. The facilitator will need to keep participants focused on the purpose of CELLA; to measure English Language Proficiency.

---

New
for 2009

Speaker Notes:

---

Addition of Binder Markings on ALL
2009 Test Books

New:

Binder Markings

(Numbers will be located on the top of the binding)

There is an arrow pointing to the top left corner of the test document

Speaker Notes:

---

Security Numbers
on Test Books Clearly Identified

Security Numbers

Speaker Notes:

---

Going Green

We’re reducing the number of printed materials:

• The Informational Flyer is available exclusively online at

• The Interpretive Guide will be printed in English and Spanish only. All languages will be posted on the CELLA website

• School, District and State Score Reports will be On CD – No paper copies

• Student Reports Will Continue to be provided in paper version

• Student Score Report has been revised to include a cut score table.

Speaker Notes:

---

2009
Schedule

Speaker Notes:

---

2009 CELLA Comprehensive
Schedule

Event:

2009 Train the Trainer for NEW CELLA Administrators

2009 CELLA Update Webinars for the experienced Administrator

Begin Online Locator Test Administration

Date:

January 13, 2009

January 15, 2009

Tuesday January 20

Wednesday January 21

February 16, 2009

Speaker Notes:

---

2009 CELLA Comprehensive
Schedule continued…

Event:

Receipt of 2009 Training Materials

Districts Begin Training

Receipt of 2009 CELLA Testing Materials

2009 CELLA Test Administration Window

Date:

By January 31, 2009

February – April 2009

March 30 – April 10, 2009

April 14 – May 15, 2009

Speaker Notes:

Refer to the front cover page of the Test Administration Manual for a complete listing of the 2009 schedule.

---

Test Security

Speaker Notes:

---

CELLA Test Security

• All CELLA testing materials are to be secured

• Test administrators must be able to account for ALL materials assigned to them

• Test administrators will be asked to sign a security form at each school site

• Test administrators will be asked to sign materials in and out each day

Speaker Notes:

Facilitator Notes:

Review slide; highlight important points/concepts. Make sure to remind participants of the security procedures that must be adhered to for any secure CELLA Test materials.

---

Districts Can Keep Non-secure Materials

Non-secure materials include:

• Test Administrators Manual

• Speaking Training CDs (ONLY)

• Unused paper bands

• Unused document count forms

• Unused return labels

• Unused non-secure large print materials

Note: ALL Secure Material Must Be Returned to ETS.

Speaker Notes:

---

Training
Materials

Speaker Notes:

---

2009 Training Materials

• 2009 Test Administration Manual

• 2009 Directions for Administration & Scoring Guides

• Set of 6 CDs

  • 4 Speaking Training CDs, one per level

  • 2 Listening CDs (A&B and C&D are combined)

• Test Book Kit

  • Level A Student Test Book

  • Level A Functional Level Test Book

  • Level A One-on-One Prompt Book

  • Level B, C, & D Listening & Speaking Test Books

  • Level B, C, & D Reading & Writing Test Books

  • Levels B, C, & D Answer Sheet

2009 Training Manual and CD handouts will be available online

Each district will receive 2 complete sets of the materials below and their orders for additional kits and individual items

Speaker Notes:

Facilitator Notes:

Make sure all participants have training materials and review the contents of the training package that has been distributed to each participant. Make sure to remind participants of the security procedures that must be adhered to for any secure CELLA Test materials.

---

Packaging, Delivery, & Return
of Training Materials

• • Training materials

    • Will be sent to districts only

    • They will NOT be packaged by school

    • Delivered by January 31, 2009

  • Training Materials must be returned with the test administration (brown boxes)

  • Reminders:

    • Return labels will be included in the test administration shipments and NOT in the training materials

    • Training test booklets can be used if needed in the administration as overage if used and free of any marks or notes

    • NO SUPPLEMENTAL ORDERS will be processed for training materials.

Speaker Notes:

---

2009 Supplemental Order Process

Speaker Notes:

---

2009 Supplemental Order Process

• Only District Coordinators can place supplemental orders

• Orders can only be placed after the initial shipment is received and checked

• Orders will be placed using the On-line Form: Once completed, print and press submit

• Orders will be shipped within 48 hours of receipt. Orders received after 2pm are processed on the next business day

Speaker Notes:

---

Packaging,
Delivery & Return
of Test
Materials

Speaker Notes:

---

• All test books packaged in 10’s

• 2009 Test Administration Manual

• Directions for Administration & Scoring Guides

• Levels B, C & D Answer Sheets packaged in 10’s

• One-on-One Prompt Book (for Level A only)

• Listening CDs

• Pre-ID Labels

• Return Material Kit

• White boxes pre-addressed with a return shipping label to ETS

Materials Available for Test Administration

Speaker Notes:

---

Packaging & Delivery of Test Materials

• • Orders are produced based on the Pre-ID file received from the DOE, supplied by the districts in Survey 7

  • Orders are packed by school (and General Service Route if applicable) and shipped to the district for distribution

  • Delivery of materials will be from March 30th through April 10th

  • A packing list of all schools will be provided to the district

  • A comprehensive packing list of all material sent to a school will be in Box 1 of the school boxes

Speaker Notes:

---

Receipt of Test Materials

Receipt of Material

• • District box(es) should be opened and checked against the box packing lists

  • Each school should count and verify that all boxes are received using the box count printed on the outside of each box

  • Each school should open the box(es) upon receipt and verify all items listed on the shipping notice are enclosed in the shipment

Speaker Notes:

---

Accommodated Format Materials
Material will arrive in an oversized brown box

• Large-Print Version:

  • Large-Print Test Booklet

  • Large-Print Answer Document for recording responses

  • Directions for Administration & Scoring Guides

• Braille Version:

  • Braille Test Booklet

  • Standard Answer Document to record test takers responses

  • A Standard Test Booklet that matches the Braille level being administered to assist the test administrator

  • Directions for Administration &Scoring Guides

• Return all Large-Print and Braille Materials to ETS

  • All completed answer documents should be sorted by grade

  • Complete a header sheet for each grade

  • Reuse the oversized brown boxes

  • Apply the pre-addressed shipping labels that is provided

  • When packing the box, place the unused material on the bottom and any used material on the top

Speaker Notes:

---

Damaged Answer Documents

If an answer document is damaged either from the initial packaging of the documents or through handling during test administration:

• For example: torn, folded, marked by a pen, etc.

• The School Coordinator needs to transcribe all the information from the damaged answer document onto a new document

• The damaged answer document must be returned to ETS and returned in the brown boxes with the Not To Be Scored materials

• ALL DNS must be bubbled in on the damaged document (Item #26)

• The new transcribed answer document should be bundled with the To Be Scored for that grade

Speaker Notes:

---

Damaged Answer Documents (continued)

If due to a student’s illness -- the School Coordinator should attempt to transcribe the student’s responses onto a new answer document

• • If unable to transcribe, make-up date within the testing window should be arranged

  • District Coordinator must call ETS at 1-866-881-2802 and provide the security number

  • The original document should be destroyed in a secure manner

Speaker Notes:

---

CELLA
Test Material Returns

Lorenzo Murolo

Inbound Operations Manager

Speaker Notes:

---

Functional Areas

Test Administrator Responsibilities

• • Required Administration Information

  • Security Log

  • Test Material Sorting

School Coordinator Responsibilities

• • Prepare Materials for Return

  • Large Print and/or Braille Materials

  • Packing District Coordinator ONLY Boxes

District Coordinator Responsibilities

• • Materials Return List

  • District Coordinator ONLY Boxes

  • Return Material to Contractor

Speaker Notes:

---

Test Administrator
Responsibilities

What/How to prepare for the School Coordinator

Speaker Notes:

---

Test Administrator Responsibilities

Required Administration Information

• • All fields must be accurately filled out

  • Ensure all students have a Test Code Disposition

  • Notify the School Coordinator if secure material is missing

Speaker Notes:

---

Test Administrator Responsibilities

Security Log

• • Complete this log when entering and exiting the room

Speaker Notes:

---

Test Administrator Responsibilities

Test Material Sorting

• To Be Scored

  • Verify the DNS has NOT been gridded

  • Group your TO BE SCORED material by GRADE

  • DO NOT insert Answer Sheets INSIDE of the Test Books

  • Notate the TO BE SCORED answer document count BY GRADE for the School Coordinator

Speaker Notes:

---

Test Administrator Responsibilities

Test Material Sorting

• Not To Be Scored

  • Unused Scannable Answer Sheets

  • Used DNS Scannable Answer Sheets

NOTE: Any Used Scannable Answer Sheets mixed with this material WILL NOT BE SCORED

• • All Used and Unused Test Books

NOTE: Used Grade K-2 SCANNABLE Test Books should NOT be included with the Not To Be Scored, UNLESS the DNS is notated

• • Secure Listening CDs

  • Secure 2009 Directions for Administration & Scoring Guides

  • Test Administration Manual

  • Security Log

  • Required Administration Information

Speaker Notes:

---

School Coordinator
Responsibilities

What/How to prepare for the District Coordinator

Speaker Notes:

---

School Coordinator Responsibilities

CELLA Document Count Form for To Be Scored Answer Sheets

CELLA Answer Document Return Summary Form

Paper Bands

Packaging

Speaker Notes:

---

School Coordinator Responsibilities

Fill out the Document Count Form

NOTE: You must create one PER GRADE and place it ON TOP OF EACH GRADE STACK. Do NOT include Blank Answer Sheet Counts

Speaker Notes:

---

School Coordinator Responsibilities

Fill out the Answer Document Return Form

NOTE: Grid ALL GRADES TESTED (Box 4) and the TOTAL number of TO BE SCORED Document Counts (Box 5).

• • The document should be placed ON TOP of the first Grade Stack

  • Photocopy the Answer Document Return Summary for your records

Speaker Notes:

---

School Coordinator Responsibilities

Prepare Materials for Return – To Be Scored

Paper Band EACH grade SEPARATELY

NOTE: The Document Count Summary (RED) must be UNDER the paper band on top of the first grade level

GRADE 3

Speaker Notes:

---

School Coordinator Responsibilities

Package all TO BE SCORED material in the WHITE boxes

If you have MORE THAN ONE To Be Scored box, place the stack containing the Answer Sheet Return Summary in the FIRST WHITE BOX

Securely tape your box on the top and bottom

The Orange TO BE SCORED return label will

already be pre-applied to the WHITE BOX

Fill out ALL steps on the return label

• • Box counts should be labeled 1 of X, 2 of X

  • DO NOT count your BROWN boxes here

  • DO NOT PHOTOCOPY your return label

  • All boxes MUST have a return label

DO NOT place a NOT TO BE SCORED

DOCUMENT COUNT FORM (Green) in this box

Return all boxes to the District Coordinator

Speaker Notes:

---

School Coordinator Responsibilities

NOT TO BE SCORED

CELLA Not To Be Scored Document Count Form

Packaging

Speaker Notes:

---

School Coordinator Responsibilities

Prepare Materials for Return – Not To Be Scored

Fill out the CELLA Not To Be Scored Document Count Form

• • Unused Test Books

  • Used Test Books (Excluding the USED Level A Scannable Test Books)

  • Defective Documents

  • ALL DNS gridded documents

  • Secure Listening CDs

  • Secure 2009 Directions for
    Administration & Scoring Guide

Ensure NO ANSWER SHEETS

were placed INSIDE of the

TEST BOOKS

Speaker Notes:

---

School Coordinator Responsibilities

Prepare Materials for Return – Not To Be Scored

Package all NOT TO BE SCORED material in the BROWN boxes

• • Unused Test Books

  • Used Test Books (Excluding the USED Level A Scannable Test Books)

  • Defective Documents

  • ALL DNS gridded documents

  • Secure Listening CDs

  • Secure 2009 Directions for Administration

& Scoring Guide

DO NOT place any TO BE SCORED MATERIAL

in the BROWN boxes

Securely tape your box on the top and bottom

Affix the Green return label on the

BROWN box

Fill out ALL steps on the return label

• • Box counts should be labeled 1 of X, 2 of X

  • DO NOT count your WHITE boxes here

  • DO NOT PHOTOCOPY your return label

  • All boxes MUST have a return label

Return all boxes to the District Coordinator

Speaker Notes:

---

School Coordinator Responsibilities

Large Print and/or Braille Materials

CELLA Document Count Form for To Be Scored Answer Sheets

CELLA Answer Document Return Summary Form

Packaging

Speaker Notes:

---

School Coordinator Responsibilities

Prepare Materials for Return – Large Print and/or Braille Materials

Fill out the Document Count Form

NOTE: You must create one PER GRADE and place it ON TOP OF EACH GRADE STACK. Do NOT include Blank Answer Sheet Counts

Speaker Notes:

---

School Coordinator Responsibilities

Prepare Materials for Return – Large Print and/or Braille Materials

Fill out the Answer Document Return Form

NOTE: Grid ALL GRADES TESTED (Box 4) and the TOTAL number of TO BE SCORED Document Counts (Box 5) for Large Print and Braille ONLY.

• • The document should be placed ON TOP of the first Grade Stack

  • Photocopy the Answer Document Return Summary for your records

Speaker Notes:

---

School Coordinator Responsibilities

Prepare Materials for Return – Large Print and/or Braille Materials

Package all TO BE SCORED and NOT TO BE SCORED material in the ORIGINAL BROWN boxes it was shipped in

Ensure the DO NOT SCORE HEADER is CORRECTLY placed on the DNS Stack, and placed on the bottom of the box

DO NOT RETURN the Braille and Large Print material with the

Regular Shipments

Securely tape your box on the top and bottom

Affix the White return label on the

ORIGINAL BROWN box it was shipped in

Fill out ALL step on the return label

• • Box counts should be labeled 1 of X, 2 of X

  • DO NOT count your WHITE or REGULAR BROWN

DO NOT SCORE boxes here

• • DO NOT PHOTOCOPY your return label

  • All boxes MUST have a return label

Return all boxes to the District Coordinator

Speaker Notes:

---

School Coordinator Responsibilities

Packing District Coordinator ONLY Boxes

2009 CELLA District Only Check List

Packaging

Speaker Notes:

---

School Coordinator Responsibilities

District Coordinator ONLY Boxes

2009 CELLA District Only Check List

• • 2009 Required Administration Information

• • Security Log(s)

• • Seating Chart (IF Required by your district

• • Packing Slips (IF Required by your districts

• • 2009 Test Administration Manuals

• • Trainings Speaking CDs

• • Unused Level B, C, D Answer Sheets

• • Remaining:

    • Unused TO BE SCORED Document Count Form(s)

    • Unused NOT TO BE SCORED Document Count Form(s)

    • Student Pre-ID Labels

    • Paper Bands

Speaker Notes:

---

School Coordinator Responsibilities

District Coordinator ONLY Boxes

Packaging

Securely tape your box on the top and bottom

Write DISTRICT COORDINATOR ONLY on the side of EACH box

DO NOT place any return labels on these boxes

Return all boxes to the District Coordinator

Speaker Notes:

---

District Coordinator
Responsibilities

What/How to prepare for returning material for scoring

Speaker Notes:

---

District Coordinator Responsibilities

Prepare Boxes for Return

Materials Return List

District Coordinator ONLY Boxes

Return Material to Contractor

Speaker Notes:

---

District Coordinator Responsibilities

Materials Return List

Verify the school coordinator correctly labeled and numbered the boxes from the school

• • No photocopied return labels

  • Label counts reflect the CORRECT BOX TYPE

Speaker Notes:

---

District Coordinator Responsibilities

Materials Return List

Log the quantity of box types you are returning on the Materials Returns List

Make sufficient copies of the Materials Return List

Speaker Notes:

---

District Coordinator Responsibilities

District Coordinator ONLY Boxes

Open ALL boxes, review the Records of Required Administration, and notify the FDOE if ANY secure CELLA materials are missing

Verify the Security Logs are complete

DO NOT DESTROY any non-secure material, UNTIL SCORES HAVE BEEN REPORTED

Ensure NO SCORABLE or SECURE material are in the boxes

Speaker Notes:

---

District Coordinator Responsibilities

Return Material to Contractor

Contact K2 Logistics to schedule a pickup, and provide the pick up location and box counts (888-886-0780)

Designate a person to be AVAILABLE at the pickup site on the scheduled date

At the time of pickup, FAX the completed CELLA Materials Return List to ETS Customer Service (866-387-2598)

Speaker Notes:

---

2009 Test
Administration
Manual

Speaker Notes:

---

Review of the 2009 Test Administration Manual

• What’s New and Reminders on pages 1-4

• Functional Level Testing on page 15

• Test Invalidation and Do Not Score on page 32

• Package TO BE SCORED Materials for return page 44

• Required Administration Information Document, Security Log, Materials Return List, and Testing Signs

Speaker Notes:

---

2009
Directions for Administration & Scoring Guides

Speaker Notes:

---

Test Accommodations

• Section in DFA to addresses ALL allowable accommodations (begins on page 288)

  • Students with Disabilities with Current IEPs

  • Students with Section 504 Plans

• Not Permitted Accommodations

  • List of not permitted accommodations is included

• Guidance on accommodations for Deaf or Hard-of-Hearing

Speaker Notes:

---

Updates to the DFA/Scoring Guides

• Electronic Use Policy added to the first section of each DFA/Scoring Guide Level

• Clearer instructions and demonstration graphic on how to fill in the responses in the Level A test book included (graphic on page 10 of DFA).

Speaker Notes:

---

Pre-ID Labels
& Student
Demographics

Speaker Notes:

---

Pre-Identification (Pre-ID) Labels

• DOE will provide information about each student taking CELLA using data from the Florida Automated Student Information Database (Survey 7, January 2009)

Two (2) labels will be provided for each student:

1. 1. The first label is for the answer sheet for grades 3-12 OR for the Level A Student Test Book for grades K-2

   2. The second label is provided as an extra if needed

or

or

Speaker Notes:

---

Pre-ID Label Sample

The following information MUST be correct for the student label to be used:

• • • Student’s Last Name

    • Student ID Number

    • District Name

    • School Number (code)

    • Grade Level

Speaker Notes:

---

Required Demographic Information
for students that HAVE a Pre-ID Label

Even if the student has a preidentified student label, the following information MUST be provided and gridded on the Level A Student Test Book, Level A Functional Level Test Book, or on the Level B, C, and D Answer Sheet:

• • • Grid 1 Student Name

    • Grid 2 Test Administrator Name

    • Grid 3 School Name

    • Grid 4 School Number

    • Grid 5 District Name and District Number

    • Grid 14 Test Date

    • Grid 17 Listening/Speaking Test Level

    • Grid 18 Reading/Writing Test Level

    • Grid 25 Accommodations (if applicable)

    • Grid 26 Do Not Score or UNDO Do Not Score (if applicable)

Speaker Notes:

---

Demographic Information
ONLY for students that do NOT
have a Pre-ID Label

• Personal and demographic information will be collected for each student:

  • Level A, last two pages of the test book; completed by the Test Administrator

  • Level B-D, first two pages of the answer sheet; assist the student if needed

• Must Grid 1 – 23 (refer to Test Administration Manual)

Speaker Notes:

---

Receipt of Pre-ID Labels

• Pre-ID labels will be provided to each district

• Labels are packaged by grade within each school

• The labels will arrive separate from test materials

• REMINDERS

  • Do NOT use Pre-ID labels from previous test administrations

  • Do NOT apply a label over another label

Speaker Notes:

On grade level testing:

• Have participants remove the Level A Student Test Book (orange cover)

• Have participants remove the Level B, C, & D Answer Sheet

• Walk participants thru where each label is placed

Functional Level Testing Use Only

• Have participants remove the sample sheet of linking labels from the training packet.

• Have participants remove the Level A Reading and Writing book (green cover).

• Have participants remove the Level B, C, & D Answer Sheet

• Walk participants thru where & why each label is needed.

---

2009 Answer
Sheet &
Do Not Score

Speaker Notes:

---

Answer Sheet Levels B-D
Reading/Writing
Item #18 Test Form and Test Level Selection

Form 2 is pregridded Test Administrator Enters the Level for Reading/Writing

Speaker Notes:

---

2009 Answer Sheet –

Item #14, Test Date – pregridded 2009, fill in April or May bubbles. Shaded so the Test Administrator knows it’s a required field

Item #15, Date of Home Language Survey

Item #16, ELL Status

Speaker Notes:

---

2009 Answer Sheet –
Continued

Item #17, Listening/Speaking Test – pregridded Form 2 – Test Administrator bubbles in Level

Item #18, Reading/Writing Test – pregridded Form 2 – Test Administrator bubbles in Level

# 23, Exceptional Student Education (Grid Only One)

• • Arranged in Alpha Order

  • Updated Exceptionalities to match FCAT

Speaker Notes:

---

2009 Answer Sheet & Level A Test Booklets

Item #26, Do not score – contains DNS and UNDO bubbles for each content area

• • If all or part of a student answer sheet for any one of the tests is not to be scored, grid the appropriate DNS bubble(s) for each of the tests.

  • If a test has been invalidated by mistake, erase the DNS bubble(s) and grid the corresponding UNDO bubble(s).

  • If the DNS bubble is gridded for any of the tests (Listening, Speaking, Reading or Writing), that section will not be scored.

  • If all DNS bubbles are gridded, none of the tests will be scored.

  • If the DNS bubble(s) for a test has been erased and the corresponding UNDO bubble(s) has been gridded, that test will be scored unless it is packaged with the NOT TO BE SCORED materials

Speaker Notes:

---

Questions and Answers

Speaker Notes: